Why Quantify?

The benefits of quantifying: It's not what you think!

Organizations frequently rely on established risk management frameworks to guide their practices. However, many prominent frameworks, such as ISO 31000 and COSO, advocate for risk heat maps. This is despite research suggesting these tools do not actually improve judgment or decision-making. The promotion of ineffective tools by international standards presents a significant problem. Given these serious flaws, why do risk heat maps fall short, and what alternative approach can provide a more reliable understanding of risk?

Risk heat maps fundamentally fail because they foster a misleading sense of communication, inadequately represent the common reality of skewed loss distributions, and operate without a transparent underlying model for risk assessment. In contrast, quantitative methods, especially those employing loss exceedance curves, offer a more robust pathway. These methods make assumptions explicit, allow for rigorous challenge, and enable sound aggregation of risks across an organization.

Risk Heat Maps Foster Misleading Communication

A primary failing of risk heat maps lies in their inability to ensure clear, shared understanding. The use of qualitative scales, such as "likely," "unlikely," "minor," or "major," proves unreliable. These words mean different things to different people.

Consider the findings from Richards Hauer in The Psychology of Intelligence Analysis. When NATO generals assigned numerical values to common likelihood terms, their estimates varied widely. For "highly likely," interpretations ranged from 51% to over 90%. For "unlikely," the assigned point estimates spanned from 0% to over 30%. What does this discrepancy cause? It creates an illusion of communication. Team members may leave meetings believing they share a common understanding of risk, but their individual interpretations can diverge significantly.

Further studies highlight an even more troubling aspect. Researchers have observed situations where, even when a qualitative label’s intended numerical range is displayed directly next to a question, individuals sometimes provide point estimates falling outside that very range. Whether using simple labels or 1-to-5 scales for impact, these tools do not achieve their intended purpose: communicating risk unambiguously. People walk out of the room thinking they agree on risk levels when they actually do not.

Risk Heat Maps Inadequately Represent Skewed Loss Distributions

In many fields, and especially in areas like cybersecurity, loss distributions are not uniform; they are often markedly skewed. This means that infrequent, large-impact events (outliers) disproportionately affect the average.

A 2016 study by Sasha Romanowski on data breaches clearly illustrates this point. He found the median loss for a data breach was approximately $170,000. However, the mean loss, inflated by those significant, less frequent outliers, was almost 17 times greater. So, which figure should populate the risk heat map – the median or the mean? This choice can dramatically alter the perception of risk, leading to misinformed decisions.

This issue of skewness is not confined to cybersecurity. Operational research encounters it consistently, and evidence suggests it also holds true for many strategic risks. The practical result for heat maps is problematic. When one overlays a loss exceedance curve (which accurately depicts skewed distributions) onto a risk heat map, it becomes clear that a single risk event can often be legitimately plotted in numerous different cells. The placement depends heavily on which part of the curve the user decides to focus on, opening the door to subjective interpretation rather than objective assessment.

Risk Heat Maps Lack a Transparent Underlying Model

A substantial drawback of risk heat maps is the absence of a visible, understandable model supporting the placement of each risk. Why did an analyst position a particular dot in a specific box on the map?

When a decision-maker reviews a heat map, they usually have no insight into critical underlying factors. What scenarios did the analyst consider when assessing that risk? Just as importantly, what potential scenarios did they not consider? Were the likelihood and impact assessments benchmarked against actual data, historical trends, or industry experience? Without answers to such questions, the heat map offers limited actionable intelligence.

This lack of transparency can lead to heat maps becoming political instruments. Teams might, consciously or unconsciously, use the subjective nature of heat map inputs to advocate for increased budgets or to downplay certain risks, rather than to reflect an objective risk assessment. This practice creates an obstacle to genuine risk reduction efforts.

Quantitative Models Pave a Clearer Path to Risk Understanding

The true value of risk quantification extends beyond generating precise numbers. Its power lies in the rigor of its process and the enhanced analytical capabilities it cultivates within an organization. The aim is not perfection in prediction, but clarity in process.

What is the core aim of quantification? The goal is to extract the specific risk scenarios an analyst considered, understand how they were prioritized, and then clearly define the range of likelihood and potential impact for each. Quantitative models compel analysts to translate their internal, often implicit, mental models onto paper or into software. This act systematically surfaces all underlying assumptions.

These explicit assumptions—about which scenarios are relevant, their probability, and their consequences—can then be discussed, debated, and refined by a wider group of experts and stakeholders. This collaborative process helps ensure everyone leaves the room with a shared, clear understanding of the risks the organization faces.

Loss Exceedance Curves (LECs) are a particularly effective tool in quantification. Insurance companies frequently use them to communicate risk precisely. LECs unambiguously display the probability of experiencing a loss of a certain magnitude or greater. They provide a full picture of the potential range of outcomes, not just a single point.

A key benefit of LECs is their ability to facilitate the aggregation of diverse risks across an entire organization in a statistically sound manner. Using heat maps, how does one meaningfully compare a "likely but moderate" risk in a small business unit with an "unlikely but major" impact risk in a much larger one? People can invent rules for such comparisons, but these are often arbitrary and lack a solid foundation. LECs, however, provide a statistically correct method for this aggregation. This offers decision-makers the comprehensive, coherent picture of total organizational risk they truly need. They want to know: how much risk are we actually holding, and where are our biggest vulnerabilities based on data?

Furthermore, the quantification process encourages, and often necessitates, that these underlying assumptions are tested and validated by multiple experts and business people. It fosters a culture of critical examination. Even if an organization’s decision-makers express a strong preference for the visual simplicity of traditional heat maps, LECs can still provide value. The rigorous analysis and aggregation using LECs can occur "behind the scenes," and the results can then be translated back into more accurately plotted points on a heat map for presentation.

Moving Beyond Illusion to Clarity

Risk heat maps, despite their widespread adoption, present considerable issues. They often fail because risks in many domains are heavily skewed, they do not demonstrably improve judgment, and they can create a deceptive sense of shared understanding where none truly exists. When teams believe consensus has been reached, but it is based on ambiguous terms and hidden assumptions, organizations face problems in allocating resources effectively, achieving strategic alignment, and maintaining trust among colleagues.

The journey of quantification, therefore, is not primarily about pinpointing an exact, "correct" number for loss and likelihood. Its main benefit lies in making all assumptions about risk scenarios, their likelihoods, and their potential impacts explicit and transparent. This transparency allows these assumptions to be rigorously challenged, thoroughly discussed, and collectively refined. This leads to greater clarity for all decision-makers on how risk was estimated. While models and estimates will not always be perfect—organizations and individuals operate with bounded rationality, and the world is a dynamic place—this disciplined process ensures a more robust and genuinely shared understanding of the risks an organization faces. Ultimately, this fosters better-informed decisions and a more resilient enterprise.