Interpretation level and the shift in residual risk: why AI risk is rarely new

What separates AI that helps from AI that quietly fails is rarely the model. It is the kind of task the model is working on, and the structure of the situation it has been placed in. That structure was described in detail decades ago, in entirely different settings, by people who had no idea AI would one day exist. The rest of this text turns on two concepts: interpretation level, and the way residual risk shifts when AI enters a process.

Reading Taleb, Kahneman, and Tetlock over the years, I kept coming back to the same observation, which I will describe below. Taleb showed how easily traders misunderstood luck for skill, and how rarely the cost of that mistake was visible until it became ruinous. Kahneman described the way the mind quietly swaps what sounds true for what is true. Tetlock asked a different question again: why confident forecasts so often fail, and what kinds of thinking actually predict well under uncertainty. None of them wrote about AI. Over the last year, I have been putting this concepts into practice in my own forecasting and AI-assisted analyses. The more I worked with AI, the clearer it got: many concepts related to working with AI were already described, just in a different context. Most of the time the output is plausible and helpful, and checking it carefully often feels like a waste of time. The structural problem is that nothing about the situation tells you when this time is the exception.

Imagine a family arriving at the airport for a long-planned holiday. They had asked an AI assistant in advance whether they would need a visa for their destination. The answer was confident, detailed, and wrong. At check-in, they are turned back. The flight is gone, the booking is non-refundable, the trip is over before it began. Nothing about the earlier conversation hinted that this would be the one answer that mattered, and the one that the AI got wrong.

Once you can name what is actually happening, the feeling stops being unclear. It becomes a set of specific questions that can be asked, answered, and built into the design of any AI interaction. Two of those questions, in my own analyses, have turned out to matter more than the rest. They are the questions I want to spend the rest of this text on, because they are where the standard discussion about AI risk stops being useful and the real work begins.

The first question is about interpretation level.

By interpretation level, I do not mean a specific model architecture or an implementation detail. I mean how much an AI system has to step back from the raw data in front of it and decide what that data actually means. Extracting a fact from a document sits low on this scale. Summarizing a section sits higher. Cross-checking several documents, deciding what matters, generating recommendations, or producing a conclusion that someone may use to make a decision sits higher still. Each step can be useful. But each step also adds distance from the underlying data.

The pattern first became clear to me in my own AI-assisted forecasting and analysis. The more I tried to improve results by adding synthesis, correction, or meta-reasoning steps, the more I noticed the same trade-off: up to a point, the system became more useful; beyond that point, it became harder to tell whether the result had improved or just become more polished.

Public cases suggest that this is not only a private workflow problem. The domains differ, and the systems differ, but the pattern of the failure is familiar. An AI system can be highly valuable up to a certain level of interpretation. Past that point, adding another interpretive layer rarely fixes the failure mode. More often, it hides it. The output becomes smoother, more coherent, and more confident, while the user's ability to see what has actually happened gets worse.

The temptation is always the same. The system works well in most cases, but there is a residual five percent at the edges where it fails. The intuitive response is to add another layer - a synthesis step, a correction model, a meta-reasoner - to catch what the base system misses. In my own analyses, this has usually turned out to be the wrong reflex. The additional layer often does not catch the five percent. It produces more confident-sounding output on top of the same underlying evidence. The added confidence looks the same as real improvement until something resolves and the failure becomes visible.

The safer move is not always to add another interpretive layer. It is to ask whether the next layer should be interpretive at all. Once the system has reached its useful interpretation level, the next step often needs to be relocated: to a human decision, a deterministic check, a specialized tool, a source-of-truth lookup, a test, an approval, or a refusal to act. The point is not that AI should do less. The point is that past the optimum, more AI interpretation does not just sometimes make the result worse. It increases the cost of knowing whether the result has degraded.

This becomes even more important once AI systems move from assistance to agency. In a non-agentic setting, a wrong interpretation may produce a bad answer. In an agentic setting, the same wrong interpretation can trigger a sequence of actions: contacting a customer, changing a record, escalating a case, issuing a refund, blocking an account, or passing corrupted context into the next step of a workflow. The interpretation layer no longer just describes the world. It becomes part of the mechanism by which the system acts on the world. That is why the threshold matters more, not less, as AI becomes more agentic.

The available evidence suggests that this is not a property of any particular domain, and not a property that better models alone can be assumed to fix. IBM's Watson for Oncology was designed to give doctors concrete cancer treatment recommendations - one of the highest interpretation levels a medical AI system can occupy. Internal documents later revealed that it recommended incorrect treatments, and MD Anderson cancelled the project. That case is from around 2016. And yet Cisco, testing in 2026 with state-of-the-art language models, encountered a similar boundary in an entirely different field. Their Talos incident response team tested whether a language model could draft security reports. With narrow, single-task instructions for small sections of a document, drafting time fell by half, and blind quality reviews came back positive. When the same model was asked to operate at a higher interpretation level — editing a complete report, generating recommendations across the document, maintaining consistency across multiple reports in a session — it hallucinated errors that did not exist, missed real ones, produced duplicative recommendations, and contaminated content from one report into the next. According to Cisco's own account, the team tested before shipping, found the boundary, and stopped.

Two cases, two domains, a decade apart, and a similar shape of failure. In my own forecasting research, the same trade-off has appeared in smaller form: the point at which additional synthesis stops making the work better and starts making it harder to verify. Whether future models will dissolve this problem is an open question. What is clear is that ten years and several generations of model progress have not removed the need to locate the boundary.

The principle that follows is easy to state and hard to apply. There is a level of interpretation at which a given AI system genuinely solves the task it has been set. Below that level, the system often fails to deliver value. Above it, additional interpretation stops adding signal and starts adding confidence, complexity, and verification cost. The optimum sits somewhere between these two, and the work of finding it is not only technical. It is conceptual.

Locating that threshold requires asking what the task actually consists of, what part of it the AI can handle, and where the point lies at which any further adjustments stops improving the work and starts obscuring it. Teams reach instinctively for more capability, more reasoning, more layers — and the system grows past the optimum without anyone noticing, because more usually feels safer than less.

What remains after the right level has been found is residual risk. Once it is named, it can be contained deliberately: with a check, an approval, a hard limit, a source-of-truth lookup, a human decision, or a refusal to act. What cannot be mitigated has to be accepted explicitly, or the design has to be rethought. The discipline sits in seeing the threshold clearly enough to know what belongs inside the system, what belongs above it, and what should not be delegated to another layer of interpretation at all.

The second question is about residual risk, and it is the more important of the two.

The standard framing in the AI risk debate is that AI introduces probability into processes that used to be deterministic, and that the answer is to wrap probabilistic components in deterministic controls. This framing is not wrong, but it is incomplete. It treats deterministic and probabilistic as categories of process, when in fact almost no business process is truly deterministic. Credit checks, identity verification, contract approval, even password resets in their classical form - these are all probabilistic processes. SMS codes can be intercepted. SIMs can be swapped. Signatures can be forged. What made these processes acceptable was never their determinism. It was the fact that their residual risk had been compressed below an implicit threshold the organisation had, consciously or not, agreed to live with.

This changes the question. The introduction of AI into a business process is almost never a categorical shift from deterministic to probabilistic. It is a shift in residual risk on a spectrum that was always probabilistic. The honest question is not whether AI is appropriate for the task in some abstract sense. It is: how does the residual risk of the AI-supported process compare to the residual risk of the process it replaces, and is the difference one the organisation has explicitly chosen to accept?

Earlier this year, Meta rolled out an AI-assisted account recovery system for Instagram, called High Touch Support. The logic was probably to reduce support costs, resolve issues faster, and scale globally. But account recovery was never a deterministic process. It had simply sat below a tolerable risk threshold for years. Attackers worked out a remarkably simple technique. Using a VPN to match the target's geographic region, they asked the chatbot to link the target account to an email address under their control. According to Meta's own account, the AI tool itself behaved as intended — the actual failure was in a separate code path that did not properly verify the email address. More than twenty thousand accounts were taken over before the flow was disabled. Whether the proximate cause is described as a bug or as a design decision, the structural point remains the same. The mistake was not that Meta introduced probability where there had been none. The mistake appears to have been a shift in residual risk - a shift that, judging by the outcome, was not adequately contained before deployment. What is visible from the outside is that the controls that would have caught such a shift were not in place by the time attackers reached the system. By the time the issue surfaced publicly, it was no longer a design question. It was a security incident.

This is the deeper lesson hidden behind the usual debate. The interesting question for any AI application - private or professional - is not whether the technology is reliable enough. It is whether the residual risk of the AI-supported version sits where the organisation, or the user, can knowingly accept it. The failures in business AI that draw public attention today are, in my reading of them, rarely failures of the technology. They are more often failures of this question being asked too late, or not at all.

These two questions matter together: what interpretation level is right for this task, and where does the residual risk sit compared to the process it replaces? Used together, they turn the design of an AI application from a matter of intuition or vendor promises into something concrete. A senior engineer, a risk owner, and an auditor can all read it the same way. They make visible where AI multiplies value and where it multiplies risk. They identify the points in a workflow where a deterministic check, a human approval, or a structural limit needs to sit. And they show where AI can be given a lot of freedom, because the cost of being wrong is small, the output is easy to verify, and the gain in speed and reach is real.

Applications built this way do more than avoid the obvious risks. They multiply what the AI does best across exactly the parts of a process where it adds real value, and leave the parts that depend on judgement, accountability, or hard rules in the hands they belong in.

The real discipline is not deciding whether to trust AI. It is learning where trust is structurally cheap, where it is expensive, and where it should not be delegated at all.