Stop Debating AI Doom. Start Managing AI Risk.
There is a lot of talk about AI risk, but the conversation focuses on ideals and platitudes that are unlikely to be productive.
“Slowing down” AI development is incredibly hard, and we (the US) aren’t in control of it. Stanford’s HAI 2026 Index shows the US-China model performance gap as closed to 2.7%.1 Deploy less capital? Hire fewer people? I would argue besides compute power, much of the progress comes from a small number of critical people, and you can’t effectively ask them to “work slower.” And unless China agreed to do the same (and we believed them), we would accomplish nothing. The genie is in fact out of the bottle.
We can’t, however, ignore the idea that risk is real. Anthropic caught a Chinese state-sponsored campaign using Claude Code against around 30 organizations by watching account-level patterns: request patterns, sequences of tool use, and activity persisting across unrelated sessions. No single prompt gave it away.2 OpenAI has disrupted more than 40 malicious networks since February 2024 and publishes what it finds.3 Real investment in risk mitigation is already happening.
The good news is that we already know how to manage risk. As Benjamin Franklin reminded us, “an ounce of prevention is worth a pound of cure.”4 There are three primary levers: detectability, probability, and severity. The goal is to invest in, design, and implement mechanisms across all three levers. This should happen at every level: the frontier labs that build the models, the enterprises and individuals who use them, and government.
Detectability: How do we know when something bad is about to happen or is happening?
Probability: How do we reduce the probability that something bad will happen?
Severity: How do we reduce how bad something is if it does happen?
I am not an AI researcher or architect, but I do understand operational performance and risk mitigation. What I outline below is not a sufficient set of proposals for achieving AI safety. I share it not as a comprehensive roadmap but to illustrate how this framework can move the AI risk conversation toward productive decisions and actions. I will also add, without unpacking it much, that while most of the rhetoric focuses on AI acting on its own to end humanity, the more likely risk is a human bad actor using AI to do it. The International AI Safety Report 2026 splits AI risk into malicious use, malfunctions (which includes loss of control), and systemic risk.5 When it comes to risk mitigation, the actions are largely the same.
Detectability: How do we detect AI risk before it causes harm?
The frontier labs have already put a lot in place on this one, but of course they can and should do much more. So let’s consider a simple analogy: I don’t want to get caught in the rain. Detectability is having good forecast models and making sure you look at them. Of course, with my domed head, I joke that I have an early rain detector built in.
The same applies to any risk, including AI. How do we detect the possibility of risky behavior, or the occurrence of a risky act? This can’t always be about bad outputs, but instead also bad inputs. We’ve learned a long time ago that you can’t control quality at the outcome level, as Dr. Deming taught us: “cease dependence on inspection to achieve quality.”6
Frontier labs: Monitor patterns across whole accounts, not single prompts.
Enterprises: Log every AI agent action against an identity: someone is accountable. Agents actually report to someone on the org chart.
Government: One common channel for reporting AI incidents with high transparency.
Individuals: An alert on every charge, monthly check of connected apps.
Probability: How do we make AI harm less likely?
Probability is how likely something is to happen. Risk mitigation is about reducing that probability. How do I manage the probability of getting wet in the rain? I can’t control when it rains, but I can control when I’m outside.
Some tools in this category amount to error proofing, preventing unintended errors from occurring. At a low-risk level, sometimes this is just about preventing hallucinations. Of course that doesn’t work when harm is intended. Adding friction points (this is often what is meant by a human in the loop) slows things down, which might reduce the benefit but also reduces the probability of something going wrong. Human in the loop is not just about Detection, but just adding that break in the system.
Frontier labs: Tighten safeguards before the capability requires it.7 Invest heavily in defensive research.
Enterprises: Least privilege: agents read-only by default.8 Humans approve money moves.
Government: Standards for agent identity and permissions.
Individuals: Narrow permissions, leverage passkeys, and consider a credit freeze.
Severity: How do we limit the damage when AI goes wrong?
Severity is how bad the outcome is once it has occurred. You may not control when it rains or whether you’re outside, but you can carry an umbrella or wear a rain jacket. This is what airbags and seatbelts are for: if you can’t prevent an accident, you can reduce the damage.
Frontier labs: Again, invest heavily in defensive research, ban at-risk accounts, leverage new models defensively before release.9
Enterprises: Transfer caps plus a practiced plan that assumes no email. IBM’s 2023 study found that organizations with high levels of incident response planning and testing saved almost $1.5 million per breach in comparison to those with low levels.10
Government: Freeze stolen funds fast, counteroffensive until “crime doesn’t pay.”11
Individuals: A family code word12 and a manual fallback.
This basic framework will be familiar to many, as it is the foundation for FMEA, or Failure Mode and Effects Analysis. This structure has evolved over the nearly 80 years it’s been utilized13, and it’s waffled between gruesome pencil-whipping exercises to rigorous analysis tools. The advantage of the latter is not just the identification of the areas of risk, or failure modes, but through a rubric the ability to prioritize those risks and guide your invested effort. Not everything can be done at once, and there are tradeoffs between risks as well as against the benefits. If those decisions seem daunting, getting organized around a simple risk formula can be clarifying and focusing.
Risk mitigation isn’t about hiding under your desk, but about identifying the problems, breaking them down, assessing them, and utilizing the levers available to you to mitigate them. This takes many parties, including those who provide policy, engineering, process, and systems solutions. Let’s get to work!
The framework at a glance
Three levers of risk, applied by everyone who builds, uses, or governs AI.
Notes
1. Stanford Institute for Human-Centered AI, “The 2026 AI Index Report,” April 2026. ↩
2.Anthropic, “Disrupting the first reported AI-orchestrated cyber espionage campaign,” November 2025. ↩
3.OpenAI, “Disrupting malicious uses of AI,” threat report series. The count of more than 40 networks was reported in October 2025.↩
4.Benjamin Franklin, “On Protection of Towns from Fire,” Pennsylvania Gazette, February 4, 1735. Franklin was writing about fire prevention. ↩
5.Yoshua Bengio (chair) et al., “International AI Safety Report 2026,” Executive Summary, February 3, 2026.↩
6. W. Edwards Deming, Out of the Crisis (MIT Center for Advanced Engineering Study, 1982/1986), Point 3 of the 14 Points, via The Deming Institute. ↩
7.For example, Anthropic activated stricter “ASL-3” safeguards as a precaution when it released Claude Opus 4. Fortune, May 22, 2025. ↩
8.IBM, “Cost of a Data Breach Report 2025,” July 30, 2025: of organizations that reported breaches of their AI models or applications, 97% lacked AI access controls.↩
9.For example, Anthropic’s Project Glasswing gives a small group of trusted organizations access to its most capable model to find and fix software vulnerabilities. ↩
10 .IBM, “Cost of a Data Breach Report 2023“: organizations with high levels of incident response planning and testing saved USD 1.49 million compared with those with low levels. ↩
11.FBI Internet Crime Complaint Center, “2025 Internet Crime Report,” April 2026. In 2025 the IC3 Recovery Asset Team froze $679 million of attempted thefts, a 58% success rate. ↩
12.The FBI recommends a secret word or phrase among family members to counter AI voice-cloning fraud.TechTarget, December 2024. ↩
13.FMEA traces to US military procedure MIL-P-1629, “Procedures for Performing a Failure Mode, Effects and Criticality Analysis,” November 9, 1949. Source: UK Ministry of Defence, ASEMS. ↩