CISOs must reconsider how security success is measured 

New technologies and frameworks are dramatically accelerating risk discovery, with Anthropic’s Project Glasswing identifying more than 10,000 critical vulnerabilities across some of the world’s most important software systems. For organisations that still measure security success by activity alone, that might appear to be a huge achievement. 

 

However, Anthropic reports that more than 99% of those vulnerabilities haven’t actually been patched yet, leaving organisations exposed despite the increase in discovery.  And with Anthropic’s just-released Claude Fable 5 — the first public version of the same Mythos-class AI architecture that sent shockwaves through the security community — that discovery engine is about to get significantly more powerful and widely accessible.

 

This situation isn’t new, but it has been dramatically amplified by AI. The release of Fable 5 is a clear signal of where things are headed: when AI systems capable of superhuman vulnerability discovery become broadly available, the gap between what’s found and what’s fixed becomes not just an operational problem, but an existential one.

 

At the same time, our research reveals that over half of organisations carry critical security debt, with vulnerabilities remaining unresolved for more than a year. This creates sustained opportunities for attackers and increases the likelihood of successful breaches. 

 

As cyber-security leaders look to strengthen their resilience, they must reconsider how success is measured and ensure that security efforts are translating into meaningful risk reduction.  Faster scans and greater volumes of findings can look promising on paper, but they don’t necessarily indicate stronger defences. Instead, organisations should modernise how they measure security and prioritise demonstrable risk reduction over time. 

 

Security activity isn’t the same as security effectiveness

Measuring against volume-based KPIs, like the number of scans run, vulnerabilities found and alerts generated, only tracks the effort taken to increase security—not the actual outcome. These traditional KPIs tell you how needed security measures are, but not whether they are stopping anything meaningful. 

 

For example, a scan finding 10,000 low-impact issues might look productive on a dashboard, but at the same time a single exploitable dependency might have been untouched for months, presenting a critical, unresolved security risk. Board members and the C-suite see rising KPI numbers and automatically assume strengthened protection when, in fact, it could be quite the opposite. This blurred measurement line skews the reality of how security teams are tackling security risk.

 

These industry-wide tropes are inadvertently rewarding security teams for generating noise but not reducing actual risk. And with the average fix time for security flaws rising from 171 days to 252 days over the past five years, the delay to remediation quietly backlogs security risks. Those vulnerabilities hidden in the depths of the supply chain and pipeline are a ticking time bomb.

 

With security teams already stretched and struggling to find the capacity to find and fix vulnerabilities, these outdated metrics encourage a culture where security teams and CISOs look “on top of it”, right up until an old, known flaw gets exploited – at which point, it could be too late.

 

Traditional scanning can’t keep up with modern attacks 

With the rapid pace of technological advancement and the apparent rise in successful cyber-attacks, point-in-time scanning is now inadequate. It overlooks critical time factors—such as the mean time to remediate or the duration an attacker can operate undetected—which are precisely what attackers exploit.

 

Modern attacks happen in the gap between scans, with security snapshots unable to catch moving targets. For CI/CD pipelines, they are obsolete. Code changes multiple times a day and dependencies update automatically. 

 

What makes this more urgent now is that the same AI capability used to find vulnerabilities at scale is increasingly in the hands of adversaries too. Anthropic itself acknowledged, when it previewed the Mythos model class in April, that it was "powerful enough to potentially help bad actors attack public and private software." Fable 5, released this week as the first broadly accessible version of that architecture, underscores that we are entering an era where the advantage — as Anthropic warned — "will belong to the side that can get the most out of these tools." That side cannot be the attackers.

 

And nowadays, an attacker doesn’t even need to evade a scan. They just wait for the next build, commit, or dependency pull and, by the time the scan report is read, the environment it assessed no longer exists. Scanners traditionally inspect source or binaries, but not the inner workings of the build process, meaning a malicious build step can inject code after a scan has passed. 

 

This happened with the infamous SolarWinds Orion, which compromised thousands of organisations (including US government agencies) back in 2020, injecting malicious code into software updates that were then distributed to the unsuspecting customers.

 

If the build is already poisoned, then the scan is irrelevant.

 

CISOs must adapt security metrics for modern threats 

As AI continues to accelerate the pace of vulnerability discovery, organisations must ensure those insights are translated into measurable risk reduction. Cyber-security leaders should focus on metrics that demonstrate tangible security outcomes rather than simply reporting security activity. 

 

This means measuring how quickly exploitable vulnerabilities are remediated, and how long they remain undetected. These indicators provide a more accurate picture of an organisation’s security posture and help prioritise action where it will have the greatest impact. Improving security metrics in this way has considerable knock-on effects, helping organisations to make better, more informed decisions. 

 

Outcome-based measurement ultimately helps strengthen alignment between security and operational priorities and ensures resources are directed where they will make the most meaningful impact. In a world where AI is discovering thousands of vulnerabilities and simultaneously lowering the barrier for sophisticated attacks, the organisations that will be best positioned are not those that find the most flaws — but those that fix them fastest. 

 

The most successful organisations will of course be those that can identify vulnerabilities, but this must be paired with an ability to consistently reduce the risk that those vulnerabilities create. That gap between discovery and remediation is no longer a metric to monitor. It is the metric that matters.

 

---

 

Sohail Iqbal is CISO at Veracode

 

Main image courtesy of iStockPhoto.com and Alex Cristi