Security: beyond vanity metrics

The quarterly security presentation follows a familiar pattern: "We processed 2.3 million threat indicators this month." "Our team identified 1,247 high-severity threats." "We generated 84 threat intelligence reports." The board nods approvingly at the impressive numbers. Nobody asks the crucial question: Did any of this actually make us safer?

 

This is the vanity metrics trap, and it’s undermining cyber threat intelligence programmes across every sector. Organisations are optimising for quantity over quality, creating perverse incentives where CTI teams focus on ingesting more feeds and generating more reports rather than deeply understanding threats relevant to their environment. The result? Intelligence overload where stakeholders ignore CTI results because they’ve learned most intelligence is irrelevant noise, whilst actual threats that matter get buried in volume.

 

The operational disconnect

Traditional CTI metrics might look impressive in presentations, but they reveal nothing about actual value delivered. When organisations measure success by indicators processed or threats detected, they enable security theatre, these are the operations that look secure but aren’t.

 

Meaningful operational metrics should answer whether intelligence helped make better security decisions. This requires measuring:

 

Actionable intelligence ratio: What percentage of intelligence results were a credible threat that required a defensive or mitigating action - a policy change, control implementation, or blocking the threat? When organisations track this, they often discover the reality sits below 10%, despite consuming dozens of threat feeds. Restructuring programmes around quality over quantity can push this metric above 40%, but only when teams cut automated feeds and focus on curated intelligence specific to their sector threats and technology stack.

 

False positive rate: Of intelligence-driven alerts or recommendations, how many proved irrelevant or incorrect? High false positive rates train stakeholders to ignore intelligence products. When SOC teams stop reading CTI reports because the signal-to-noise ratio is abysmal, at this point you’re not running an intelligence programme - you’re actually running an information dumping ground.

 

Intelligence-to-action timeframe: Average time from intelligence receipt to action taken. This reveals whether your CTI operates at the speed of business or is trapped in bureaucratic processes. Organisations that reduce this from weeks to hours typically achieve it not through new technology, but by fixing stakeholder alignment, pre-approving playbooks, and building cross-functional relationships that eliminate decision bottlenecks.

 

Coverage efficiency: Are you generating intelligence across your actual threat landscape, or only monitoring what’s easy to monitor? Many organisations excel at tracking commodity threats whilst remaining blind to adversaries that actually target their sector or business model.

 

Threat context: Are you generating intelligence that empowers the right stakeholders to take the right action? This means ensuring that they know where the threat originates from, why it is a threat, and what can or should be done to eliminate the threat. 

 

What the board should demand

Board-level CTI reporting typically focuses on easily quantifiable but ultimately meaningless metrics. What executives should demand are business-relevant KPIs demonstrating whether CTI investments reduce risk and enable better decisions.

 

Risk-adjusted decision support: How many business decisions, mergers and acquisitions, vendor selection, market expansion, etc, were informed by threat intelligence, and what risk did this mitigate? When organisations evaluate acquisitions, threat intelligence should analyse the target’s exposure and whether their systems are actively targeted. This intelligence doesn’t kill deals; it informs post-acquisition integration requirements and risk-adjusted valuations.

 

Threat coverage versus threat exposure: What percentage of your actual threat landscape are you actively monitoring? Where are your blind spots? Rather than claiming comprehensive coverage, mature programmes honestly assess gaps. "We’re monitoring 73% of our identified threat landscape, with current blind spots in IoT device threats and supply chain attack vectors" tells leadership far more than "we monitor all threats."

 

Intelligence-driven versus reactive response: What percentage of security incidents were anticipated by intelligence versus discovered through reactive detection? This metric reveals whether CTI is genuinely providing early warning or simply documenting threats after they’ve impacted you. Organisations that track this often discover they’re far more reactive than they believed. Improving this metric requires shifting from consuming commercial feeds to proactive hunting - be that monitoring domain registration patterns, adversary infrastructure, or threat actor communications before attacks materialise.

 

Cost avoidance through proactive defence: Estimated cost of threats prevented through intelligence-driven pre-emptive actions. This is admittedly difficult to measure precisely, but even approximate figures matter. When CTI analysis leads to delaying market expansion until specific controls address active APT activity in a region, quantifying that avoided risk demonstrates value in language executives understand.

 

Moving from theatre to substance

The annoying reality seems to be that most organisations measuring vanity metrics create the illusion of security without demonstrating actual risk reduction. They can tell you about the latest malware variant but can’t answer whether expanding into Southeast Asian markets increases their exposure to specific nation-state threats.

 

Breaking this pattern requires uncomfortable honesty about what your CTI programme actually delivers. It means accepting that processing fewer indicators whilst deeply understanding relevant threats delivers better outcomes than ingesting everything. It means restructuring programmes to allocate effort not just to operational detection, but to strategic intelligence that informs business decisions.

 

When CTI teams measure what matters, actionable intelligence delivered, decisions informed, blind spots identified, response times reduced, they transform from cost centres explaining why they need more threat feeds into strategic enablers demonstrating how intelligence reduces risk and enables safer business innovation. That transformation doesn’t require new technology. It requires new metrics.

 

---

 

Graeme Huddy is a Director at Mobius Binary

 

Main image courtesy of iStockPhoto.com and Boy Wirat