If you’ve read some of my previous posts, this might sound a bit familiar. This is part of a recurring theme about measuring security, which, let’s face it, we don’t do well enough. So the issue is this; you’ve invested in a bunch of cybersecurity controls and processes, but do not really know how to measure your efforts.
What’s common is to define and track certain metrics, KPIs (Key Performance Indicators), and KRIs (Key Risk Indicators). But how do you make sure these are relevant (to quantify risk events), mature enough, and up to date? Many of them will become obsolete over time or would require a new way to measure them. Continuous monitoring and adaptation is the only viable strategy there. But that’s easier said than done given the current level of maturity in both understanding and assessing risk, and also in defining metrics (in addition to measuring them and their implications, or in defining associated tolerance levels or thresholds).
Metrics: key factors to derive risk
There are quite a number of metrics that people are using as a way of keeping tabs on their security efforts. These would also be the elements that influence the likelihood and the impact of a risk event, such as the risk associated with a vulnerability or a system weakness exploitation for instance.
From a technical perspective, measuring the difference between using a given set A vs. another set B, and/or any associated thresholds, is almost impossible. There isn’t a ground truth so to speak to most of this, as it depends on each and every environment. Also, AFAIK there isn’t a straightforward and generic way to give a security rating or implication to any of these metrics. Here is an example set of categories and metrics.
| Cyber hygiene and posture metrics | How well you’re tracking and documenting vulnerability assessment/ patching, and compliance; e.g.: – % of systems with regular vulnerability and risk assessments – % of systems covered with tested security controls – % of non-compliant systems (wrt a given rule or benchmark) – No. of identified risks and severity – No. of identified exploitable vulnerabilities – No. of identified exploited vulnerabilities – Remediation and update rates per technology and vulnerability type – No. of policy exceptions – % of potentially unidentified devices on the network |
| Time/ reactiveness metrics | Time to events; e.g.: – Mean Time to Detect (MTTD) – Mean Time to Acknowledge (MTTA) – Mean Time to Contain (MTTC) – Mean Time to Resolve (MTTR) – Mean Time to Recovery (MTTR) |
| Cyber awareness & Training metrics | How well you’re tracking and documenting who in your organization has taken a piece of training, and are you making sure they understand the material. |
| Access management metrics | How well you’re tracking and documenting who has access to what. |
| Historical cyber hygiene metrics | How well you’re tracking and documenting your security controls findings over time; e.g.: – No. of phishing attacks identified – No. of virus infections identified – No. of virus attacks blocked – No. of spam blocked – No. of intrusion attempts – No. of port probes |
| Monitoring metrics | How well you’re tracking and documenting the traffic/ systems usage and possible anomalies. How well you’re tracking and documenting threat intelligence (e.g. IP reputation, hacker chatter, leaked credentials, etc.). |
| Test performance metrics | How well you’re tracking and documenting findings in attack simulation (e.g. pen-testing, BAS). |
So how do you address real-time risks?
Many of the above metrics are rather static assessments, which will likely fall short in efficiently quantifying risk events in today’s dynamic threat landscape. Obviously, not all of them require constant updates, but that’s not even the main issue IMHO. I believe most security and risk practitioners recognize the threat landscape as highly dynamic, but only a few pay attention to the intricacies of interactions between threats and the system under consideration. These are interconnected in non-trivial ways, i.e. not only a new threat observation might require an adaptation wrt metrics, but also an operation on the system itself, independently of the threat landscape, e.g. deploying a new technology or even an update or patching of an existing one, which might carry a certain risk on itself. Folks who have done threat modeling or manual assessments have a better understanding of these subtleties.
I have recently published a paper about the idea of adaptive risk assessment (Secrypt 2020), which can provide the flexibility needed to deal with real-time dynamic risks. It still, however, needs a more comprehensive knowledge representation of the cybersecurity domain (which remains one of the major shortcomings in cybersecurity), and an analytical model that allows for a holistic view of risk, from prevention through response and recovery. Defining the right metrics is only one small step towards well-planned risk assessment. One would still need to identify potential risk events and quantify them based on chosen metrics, while continually identifying opportunities to minimize risk by acting on critical indicators (as shown by them metrics). Ultimately, adapting to change matters, in both metrics and risk and threat modeling.
The need to adapt is not new. I’m not breaking news here, but people and industries with adaptive abilities have often been more successful in the past. There isn’t anything more dangerous for security than continuing to do the same things over and over again. Intellectual inertia can be very difficult to combat. My message here is simple; choose your metrics with care, a set that is actionable and meaningful to your company and what’s happening out there. And above all, the evaluation and assessment process should always be able to adapt to changing conditions (via feedback loops, collaborations, informed prediction or projection, etc.). For me, these are the attributes of a strong risk program.
