The importance of visibility about the decisions AI is making

The more deeply we work with AI within security operations, the more convinced I am that visibility will determine how useful it becomes to the SOC.

 

This does not mean another dashboard or another layer of telemetry. Most SOCs already have more information than an analyst can consume, and adding more data is hardly a measure of progress. 

 

What is more interesting, and critical, is when AI influences an investigation, whether the analyst can see what it has drawn together, why the assessment has changed and why one course of action was prioritised over another.

 

These questions become quite real inside a SOC. If AI gives an alert a higher confidence score, knowing the score is not enough. If an investigation moves in a different direction, the analyst needs to understand what changed. 

 

Furthermore, when AI recommends containing a user or isolating a device, the team needs enough visibility to decide whether that action makes sense in the circumstances they are dealing with.

 

Most of the conversation surrounding AI still starts with capability. How much can it do? How quickly can it analyse data? How many investigations can it accelerate? 

 

While these are important, another important question is how we make that role genuinely useful to the people who still need to understand the risk and decide what happens next.

 

AI visibility

Explainability, observability and visibility are frequently used when discussing AI. However, in security operations, visibility needs a much more practical meaning.

 

Analysts must be able to follow the journey of an investigation. What evidence did the AI consider, what context did it introduce, and why did that context alter the assessment? More importantly, how did the assessment influence the decision or action that followed?

 

This does not mean analysts need to understand every mathematical operation inside a model. What they need is enough of the operational decision trail to understand how the technology is influencing the investigation in front of them.

 

The amount of visibility required will inevitably depend on the influence AI has. If it is summarising an alert, the consequences of limited visibility may be relatively contained. Once it begins prioritising investigations, recommending containment or initiating an action, the expectation has to change.

 

The greater the operational influence, the more the SOC needs to see.

 

When decisions become obscure

There is a tendency to frame AI risk around whether the technology might be wrong. Of course this will inevitably happen occasionally.

 

The difficulty comes when we cannot properly interrogate how a decision was reached.

 

With a human decision, we can normally ask what information was available, what assumptions were made and why a response appeared reasonable. It may expose poor judgement or a flawed process, but there is something to examine and learn from.

 

If an AI-assisted decision is obscure, the conversation is different. Without visibility of the evidence, context and rationale, the team is left deciding whether to trust the output or reject it. Neither option is good enough for a mature security operation.

 

This is where concerns around black-box AI sit. It doesn’t mean organisations must demand proprietary source code, nor that analysts interpret the inner workings of every model. The SOC needs to see what informed a meaningful decision, where uncertainty existed and whether AI recommended an action or actually took one.

 

Without that, dependency can creep in almost unnoticed. Recommendations are useful, analysts become comfortable with them, and processes gradually form around conclusions that people no longer routinely interrogate. The technology may have become opaque, but the team may also lose the habit of asking why.

 

Designing visibility in the SOC

For meaningful AI-assisted decisions, organisations should decide what the analyst needs to see. Evidence and contextual sources should be identifiable, and recommendations should be distinguishable from actions the system has executed. If an analyst intervenes, overrides the AI or reverses an action, that should remain part of the record.

 

None of this requires an enormous AI dashboard. 

 

The practical test is what happens during an investigation. Can the analyst understand why an assessment is developing in a particular direction while there is still time to question it, validate it or choose a different action?

 

I would then go back later and ask whether somebody can reconstruct what happened. If a significant AI-assisted decision needs to be examined after an incident, the team should be able to trace the evidence and context, understand how the investigation progressed and see where human judgement changed its direction.

 

Finding the decisions AI is already influencing

For organisations trying to improve visibility,  start by looking at where AI actually influences security decisions rather than producing an inventory of every tool with an AI feature.

 

One product may advertise dozens of AI capabilities while changing very little about an investigation. Another may quietly influence alert priority, incident severity and analyst behaviour throughout the day. Those are the points that should be understood first.

 

Once you know where AI has influence, you can ask whether visibility is proportionate to the consequence of the decision. Then test the reality. Take an AI-assisted investigation and ask the analyst to explain what the AI saw, what changed and why the investigation moved as it did. Take an older decision and see whether the team can reconstruct it.

 

The gaps will become apparent quite quickly. My suspicion is that some organisations will discover AI already has more influence over their security decisions than their operating processes recognise.

 

Identifying an AI-powered SOC provider

The most important question to ask an AI-powered SOC provider is what analysts are going to see.

 

Highlight an alert the AI has suppressed, escalated or acted upon. Provide the evidence it considered and what changed the assessment. If there was uncertainty, where can the analyst see it? If AI recommended an action, is that different from the system executing it? If an analyst disagreed, where is that intervention recorded?

 

Most importantly, demonstrate what the analyst could see at the moment they were expected to make a decision.

 

This demonstration will show more than a presentation about the intelligence of the model.

 

AI is going to influence more investigations and, in some environments, take a greater role in operational action. This is critical because of the pressure analysts face today, 

 

But greater capability should not require us to accept less understanding.

 

We have spent years asking security teams to become better at explaining risk, showing their decisions and creating accountability around operational action. It would be a peculiar step backwards if, just as the technology becomes more capable, the decisions inside the SOC become less visible

 

For me, mature use of AI in security operations is not defined by how much we allow it to decide; it’s whether the people protecting the organisation can still see enough to understand those decisions, question them when something does not fit and act with confidence when it matters.

 

---

 

Keven Knight is CEO of Talion Cyber Security

 

Main image courtesy of iStockPhoto.com