The Butler Did It: Why We Shouldn’t Automatically Blame the Model

In old-fashioned detective stories, the murdered millionaire’s butler is the easy suspect: he’s always in the room, knows the family’s secrets, and has quietly detested the victim for years.

Likewise, when things go wrong in an AI system, the model is often the first suspect. Due to the hype AI models receive in the media, people often give their abilities too much credit and, conversely, assign them too much blame when systems fail.

To give a real-world example, one of our clients saw a high degree of variability in employee performance using an AI tool. Some workers doubled their output (with slightly improved quality), while others achieved only a modest 25% increase.

The client consulted their internal data scientists, who suggested looking at the “temperature” settings on the AI models involved. On one level this would seem to make sense: temperature is one of the settings that controls the variability of AI outputs (on the level of whether it says “persimmon” versus “apple” when the user says “name a fruit”), so if you’re seeing variation in the performance of an AI system, perhaps temperature is to blame?

However, in this case there was another suspect hanging around the scene of the crime: employees’ work process.

Without divulging too much detail about the client (call it “witness protection”), when we examined the transcripts of sessions we found that some employees with the lower productivity multiple were actually putting in the same number of hours on the task that they did without the AI assistant. Before, they could barely keep up with the incoming work items they needed to process. With the AI assistant, they not only caught up with the backlog, but actually re-processed items from the previous batch.

Basically, the idea of “OK, you’re all caught up since last time, stop working and do something else” was inconceivable to these workers after decades of conditioning, so they just kept going. But those who adapted saw the 2x productivity gain and did stop and do something else, boosting output for other stages of the workflow that still required manual work.

This explanation certainly fit the available evidence better than the temperature theory: the actual output of the AI model was not appreciably different between the highly productive and less productive groups. But whether we were checking the temperature settings or reviewing how employees actually used the tool, having detailed logs available was crucial. The difference was where we looked.

Alan Moore, the acclaimed graphic novelist, once said that his Victorian murder mystery From Hell wasn’t really about Jack the Ripper so much as the societal ills of 19th-century England that created the slums where Jack the Ripper preyed. If you truly wanted to solve those murders, he said, “you wouldn’t just have to solve the crime… you’d have to solve the entire world that that crime happened in.”

That’s a pretty good description of what it takes to explain an AI system’s behavior. The thing producing a response and a performance outcome isn’t just an AI model. It’s the model plus its instructions, the knowledge it can access, the routing logic, the workflow, the interface, and, crucially, the humans using it.

Interrogating the Witness: Why You Can’t Just Ask the AI

Of course, sometimes the model really is a person of interest. But even then, your conclusions will vary depending on how you analyze its behavior.

I’ve seen plenty of intelligent business executives and researchers ask an AI agent how it arrived at an answer or cite its sources for that answer, after the fact. The problem with this is that AI models don’t learn from experience the way humans do. 

Before developing AI systems my company created training programs for human workers doing highly technical jobs. One of our favorite tricks to help learners activate their prior knowledge of a subject was to state where they might have learned it.  For example “as I’m sure your professors taught you in your organic chemistry class at university” or “when you were in boot camp I’m sure your drill instructor told you to do XYZ…” Evoking memories of specific people and places can resurface any abstract knowledge they acquired through those experiences.

AI models do not share this connection between abstract information and experiential memory. Once a given pattern of information has been burned into its “weights” (the set of statistical parameters that determine how AI models make connections between words and concepts), the model can’t remember exactly how it gained that knowledge. And unless it followed some kind of structured decision making process, documenting its reasoning at each step, the AI model can only guess at what led it to say “Shanghai” when the user asked about cities to visit in Japan. The model’s guess is about as good as yours. 

So when you demand that an AI model explain its statements, absent a larger system for auditing you’re likely not getting a definitive accounting.  Any response it provides under those conditions is like the classic unreliable witness cracking under interrogation: not lying, exactly, but telling the detectives what it thinks they want to hear. You get a plausible story, not a record of what happened.

This is not a uniquely AI problem: it’s actually uncomfortably close to what humans do. My wife and I met at a friend’s house 30 years ago. She claims we had our first conversation in their living room; I remember it taking place in the kitchen. We’re both sincere, and at least one of us is reconstructing the past rather than replaying it. The one detail we both agree on is that I was conspicuously not wearing socks.

Airtight Alibi: Documenting AI Reasoning

Any good compliance attorney will tell you the time to document a decision with safety or ethics implications is when you make it, not when the subpoena arrives. The same goes for AI: have the system document its every move as it goes. Explicitly instruct the system on how to reason through your task (e.g., 1. identify the applicable policy, 2. list the relevant evidence, 3. check for exceptions, 4. recommend), and have it do that reasoning “out loud” printing its output for every step before delivering a response. Now the audit question becomes much more tractable: did the system follow the process?

Now, some of you reading this will point out that most AI labs will provide records of a model’s “reasoning trace” (the rough drafts a high-end model produces before generating a visible response) if you flag it in your requests to their platform. Anthropic has even begun publishing interesting details about what researchers call “J-space”: a small internal notepad of key words and concepts the model actively references while reasoning.

However, those records should not be accepted uncritically. The reasoning traces that frontier labs display alongside their models’ answers are often deliberately obfuscated summaries of the actual reasoning drafts, to prevent competitors from distilling their closely guarded reasoning frameworks. This makes the traces interesting, maybe even useful, but not definitive as evidence: at best, they are a form of digital hearsay. And while J-space research could one day improve AI explainability, it’s still highly experimental, and it offers only a partial view: the notepad holds a few dozen concepts at a time, and much of what the model does never passes through it.

There’s also another reason not to abdicate explainability to a model’s built-in reasoning: while the AI labs have invested a great deal of time, money and effort into refining their models’ default reasoning steps, they won’t necessarily mirror what your organization would consider best practice or compliant, especially if you work in a highly technical or regulated domain. In those cases, it’s better to hand the model your own framework, and demand that it share its reasoning out in the open before arriving at a conclusion (and, depending what platform you use to build your AI agents, you can still hide that chain of thought from the user in the final output.)

Examining the Crime Scene: A Systematic Approach to Audit and Explanability

So how should organizations audit AI system behavior in high-stakes settings?

The answer comes back to Alan Moore’s notion of solving the entire society (or system) in which a crime (or an incorrect / undesirable output) occurred.

This sounds daunting, but in practice it starts the way any detective reconstructs a crime: by retracing the steps. What went into the system? Who touched it along the way? Where was it routed, what knowledge did it draw on, what came out, and what did people do with it afterward?

For high-stakes processes, this often means minimizing how much is left to the AI model’s discretion, or the user’s. Picture a sales advisor agent instructed to walk the user through a consistent process – leveraging structured information about a customer’s requirements, budget, and procurement policies – and requiring the salesperson to give complete input where required, versus the same salesperson giving a general purpose AI chatbot two sentences of context about a deal then asking “How do I convince this customer to sign?”

Basically, you can either make the system smarter (give it a more coherent operating philosophy, so it recognizes inadequate input and asks follow-up questions or refuses to proceed) or make it dumber (replace the free-text box with a form and a few dropdowns) – the second option being wildly underrated.

In both cases we want to capture comprehensive logs, but in this case it’s more about having a structured process and tracking process compliance (i.e. did the AI model and/or the human do what we told it to do?) than trying to read the statistical tea leaves inside a model’s J-Space or temperature settings. 

And then there’s a larger question of process validity – if the AI system and human users follow our instructions faithfully, will it actually produce the outcomes we want? If an AI agent advising salespeople on procurement negotiations is executing on a bad strategy, who cares whether Gemini or Kimi is running it? Poring over session transcripts and highly granular model parameters won’t lead us to the improvements we want.

Gathering Evidence (While Protecting the Innocent)

None of this is an argument against detailed audit trails. The platform my company markets captures just about everything: every model that generated every message and all its settings, what data sources were loaded into its context at any given moment, whatever hidden reasoning was generated, the reasoning traces and J-Space data if available, how many tokens it burned, how many milliseconds it took, and so on. And I obsessively monitor our own AI systems, especially the ones that run at high volume or high-stakes settings with a low margin for error or a thin economic margin, chasing that next 1% improvement.

Sometimes all that detail is invaluable, just like fingerprints and DNA evidence in detective work. 

Once, we built a system for a client who needed news analysis on events affecting their industry. We instructed the agent to view news sources critically: audit every claim, and consider each source’s known biases and editorial standards. It worked well – for a while. Then Anthropic introduced stronger safeguards against misinformation into its models, and those safeguards interacted with our existing instructions to create a pathological paranoia about news sources, to the point where the system simply refused to comment on anything it was fed.

Without model version tracking, we might have concluded “our prompt stopped working” or “the agent has become unreliable.” Instead, the audit trail showed us that an external dependency had changed underneath us, and we re-engineered around it. Sometimes the most valuable question an audit trail answers isn’t “Why did the AI think this?” but “What system actually existed at the moment this happened?”

That said, when a client tells me “I want to look at X,” I sometimes feel like Sherlock Holmes listening to his arch-rival Police Inspector Lestrade preparing to arrest the wrong person. For instance “How long did people spend working on their sales proposals – I want to make sure they’re not being lazy, letting the AI do everything!” 

While there’s nothing wrong with capturing time-on-task data, in that particular case you have to ask “Why not look at the output, and whether it’s acceptable? Ideally, if the AI system is doing its job, it should take less time…”

Once again, even with insanely detailed logs available, you have to be careful what you share with different stakeholders in reports and dashboards. Sometimes, withholding or redacting information leads to better conclusions than making everything available to everyone.  And you also have to consider whether the audience will know what to do with your audit and explainability data: a 2025 paper from MIT Lincoln Laboratory, “Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans,” reviewed 18,254 papers on explainable AI. Only 253 even used language suggesting humans were involved in evaluating the explanations, and only 128 actually ran some form of human study. That’s 0.7%.

In other words, saying “We captured logs, that solves the explainability problem” is misleading. The larger question is whether humans find the explanations useful, and their impact on decision making. Do we care whether a doctor says an AI system’s explanation feels credible, or whether its explanations help doctors catch more diagnostic errors?

Whether explanations actually improve decisions is better measured with the systems we already have in place for human performance.

Case Closed?

If you’re an individual using a chatbot to draft emails or brainstorm, none of this matters much. If an answer seems off, you hopefully notice, shrug, and edit the AI’s output or ask again.

But if you’re deploying AI into consequential workflows at enterprise / institutional scale (clinical guidance, regulatory compliance, multi-million-dollar negotiations, financial reviews), explainability and auditability are essential. So audit, absolutely. But do it holistically:

• Build for explainability from the start. Design workflows with explicit reasoning steps and capture everything, even if you won’t look at most of it most of the time.
• Investigate the whole scene. That includes the context, the routing, the interface, the process, and the humans, not just the model.
• Know your motive. Measure at the level where consequences become meaningful, and save the forensic deep dives for when something breaks or when a lot depends on a single component.
• Show the right evidence to the right jury, and judge the system against the human performance it’s replacing, not against perfection.

And when something does go wrong, resist the urge to round up the usual suspect. 

The butler is often innocent.

R.I.P. Tim Curry, still the world’s greatest murder mystery butler.

Also, Kermit the Frog was clearly aware of regulations on soap advertising.