When used effectively metrics can help identify strengths and weaknesses in controls and processes in an organisation’s cyber security program and provide a sense of the value being derived from it.
Not only can metrics measure how well a security program is doing, they are important when it comes to communicating results and overall progress to the C-suite. However, metrics are not an exact science and a key challenge that many cyber security teams have is finding and gathering the right metrics.
Added to that, often the metrics that security organisations track and present to management are not aligned with business objectives. They tend to be too focused on compliance and do little to convey how effective a security program is in reducing overall risk.
To cut to the chase…. metrics are very useful, but only if they track the things that matter. So which metrics aren’t worth the hassle, and which ones are? To find out teiss pulled together a panel of experts to draw on their experiences of metrics that do and don’t work.
Number of threats blocked
Tim Bandos, VP of Cybersecurity at Digital Guardian said, "My favourite ineffective security metric that I’ve encountered is ‘Number of Threats Blocked by Security Controls’. Of course, it sounds amazing to report to the board that your controls blocked millions upon millions of threats at your perimeter firewall, but anecdotally this is the absolute worst.
"It sends the wrong message in relation to the effectiveness of your cybersecurity program and doesn’t truly gauge how resilient your organisation is to an actual threat such as ransomware or a state-sponsored attack.
"A better metric in my opinion here is the mean cycle time from initial infection to detection or the duration to neutralise a successful threat, because at some point, they will get in!”
Josh Flinn, Director of Product Strategy & Innovation at Cybera, had a different view on the value of this metric, however.
He commented, "All information available in security metrics is useful. Some metrics, like the number of threats blocked, seem less useful, however a spike in this metric can indicate an active attack against your network or a compromised endpoint.
"Attacks are becoming more sophisticated so the more information you are armed with the better. The big problem with security metrics is the vast amount of information that is available now. Sifting through all the data and trying to correlate it is more than a person or team can reasonably do.
"The key for security going forward is AI and ML, so the security professionals can focus on the threats instead of the data."
Richard Cassidy, Senior Director Security Strategy at Exabeam, told us that he thought that MTTD (mean-time-to-detect) and MTTR (mean-time-to-response) far too often focus organisations on ‘alerts’ and how quickly security teams can triage, close or escalate them.
He said that, “In essence, this is a kind of ‘alert-whack-a-mole’, except it’s an infinite game that results in ‘alert-fatigue’, which (as the industry breach metrics prove time and time again), simply leads to even poorer security outcomes.
“We should turn our attention to metrics that tie security to business context; there’s a new concept to consider – mean-time-to-answer (MTTA). Technology has now caught up to enable a much more context enriched story of the chain of events, as they relate to a user (be it an exec or privileged user) and a critical asset (be it a database, server or key host).
"We’ve got to start focusing security and GRC teams on how they can provide better ‘answers’ on the risk or threat context of an alert, so that we drive a far more relevant and business critical outcomes."
Prioritising technical certifications – an outdated approach
According to Matthew Buskell, Area Vice President at Skillsoft, treating technical certifications as the priority recruitment metric is an outdated approach.
He explained, "It’s time for firms to demonstrate a greater willingness to diversify their workforce and assess what traits are required — lateral thinking, problem solving skills, an understanding of risk management — rather than narrowly focusing on technical certifications alone. This requires a depth and breadth of vision that goes beyond traditional thinking.”
Buskell explains that this approach can have a positive impact on reducing the gender gap in the security industry. He continued, “When it comes to mining the potential of the female empowered workforce, numerous national programmes are encouraging women to acquire cyber-skills.
"The UK’s National Cyber Security Centre has created courses to encourage girls to consider studying the subject at A-level and university. Similarly, since 2013 the Code First: Girls organisation has been supporting young adult and working age women in the UK to develop further professional skills, such as coding and programming, and working with companies to help them capture top female tech talent."
Not paying attention to what’s lurking in the shadows
Michael Scheffler, AVP EMEA at Bitglass, talked about cloud security.
He said that in this instance “one of the least useful metrics is to measure the number of cloud services that employees are using.
"This is because as much as IT security teams like to think they are blocking all cloud services not approved by them, the stark reality is that there are likely hundreds of cloud services being used by employees that the IT security team has no knowledge of. Referred to as shadow IT, this poses security and compliance risks since sensitive corporate data is stored in shadow IT cloud apps – yet the company has no control over that data.
“The second security metric that simply doesn’t work is to assume that traditional tools for safeguarding data on premises are equally capable of protecting data in the cloud.
"With more and more organisations storing sensitive information in the cloud – information like customer data (45 percent), employee data (42 percent) and intellectual property (24 percent) – adopting proper cloud security measures is critical (source: Guardians of the Cloud, Bitglass’ 2019 Cloud Security Report).
"Over the last five years, cloud adoption has grown at an astonishing rate. Consequently, employees have been able to work more efficiently and flexibly, allowing organizations to enhance their operations in various ways.
"With that being said, the need for data protection is more vital than ever, and security strategies that organizations are implementing must be shaped around a cloud-first environment. Adoption rates of basic cloud security tools and practices are still far too low – and many organisations need to rethink their approach to protecting data in the cloud.”
Qualitative doesn’t translated into quantitative
Chief Security Technologist at Node4, Steve Nice, said that, from his perspective, “trying to turn qualitative evidence – how we ‘feel’ about a situation for example – into a number between X and Y is fine, but knowing what the ‘value’ actually is, is entirely arbitrary.
"The point of exercises and metrics like this are to simplify comparison – nothing more. But as soon as you try and perform anything more the most facile comparative analysis on them, their use and meaning become a hindrance rather than a help. So, terms such as ‘average’ in this context are actually deceptive.
“When it comes to ‘value’ in management terms, I think as long as we’re consistent with regards to how we arrive at it, and avoid the temptation to ‘over-reach’ on its statistical significance, it could be a useful metric. The crucial plot in all of this, is if we make a statement like “the threat has decreased from ‘X’ to ‘Y’”, we can explain unambiguously what it was in the various composite terms/situation that initiated this change.
“Essentially, I don’t think it’s really that important whether we have a security status set in ‘blue’ or ‘pink’ … it’s more to do with whether we’re ‘bluer’ or ‘pinker’ than we were last time we looked; why this is so, and what we need to do to make things better.”
Number of critical vulnerabilities patched
Tim Bandos later added that another favourite ineffective security metric he’d seen get reported is just showing the ‘Number of Critical Vulnerabilities Patched’. Bandos said, “Yes, it sounds great to say to your CIO that you’ve patched 100 critical vulnerabilities.
"However, let’s say your environment still has 1,000 outstanding vulnerabilities that still need to be patched – and by the way, those vulnerabilities exist on some of your more critical infrastructure that house sensitive data.
"Don’t get me wrong; I still think it’s crucial to demonstrate your progress in patching, but we need to avoid only showing the positive and provide a more comprehensive view of risk and measurement of your program’s effectiveness."