Observing the Unseen: The True Measure of Success for Security Awareness Programs

If the strategic goal of a security awareness programme is to change user behaviour on an organisation-wide scale – essentially, making good cyber hygiene behaviour a self-regulating requirement for belonging to the organisation’s culture – how do we know when we’ve achieved our objective? This is a vexing challenge for those of us in the field. It’s far too easy to miss the forest for trees.

When you programme is new, your goals are tactical and limited. You might track the percentage of users who complete their on-boarding security training. Something that’s simple to measure and roughly indicative of program effectiveness. As your programme matures, you flirt with metrics and statistics, via one-off reports and comprehensive dashboards. How many users “fell” for a phishing simulation?

How many departments have a “security ambassador” appointed (and do they show up for meetings)? What percentage of company laptops can be validated in a quarterly inventory? Most awareness pros will grab hold of whatever concrete numbers they can assemble to convince upper management that they’re doing something well … but those arbitrary “success criteria” graphs don’t provide any insight into the cultural impact of the awareness programme. In short, are people doing what’s needed because they choose to? 

I suggest that the best way to measure programme-level impact can’t be tracked with mathematics; it can only be observed on the office floor. To recognize wilful compliance, you need to silently observe to how users act when their only motivation for doing “the right thing” (security wise) comes entirely from within themselves. 
As an example, I was gobsmacked week-before-last during a lousy breakfast. I was out-of-town, attending some family medical drama. The details aren’t important to this story; what matters is that my wife and I had stopped at a popular restaurant before meeting our relatives. We’d chosen a place that we’d frequented in the 1990s on long trips (but hadn’t been to in years). 

We were fatigued from too many early mornings, late nights, missed meals, and unrelenting stress. With no small children to entertain, we each tended our beverages in companionly silence. My wife was concentrating on steeping her tea. I was focused on getting coffee into my muzzy head as swiftly as possible. While I drank, I scanned the room, curious to observe the locals.