An AI Explanation Is Not the Mechanism

An AI system gives a recommendation, classification or answer. Then another box appears beneath it: Why? The explanation is fluent, specific and reassuring. It names several inputs, describes a chain of reasoning and sounds like the kind of account a careful person might give. None of those qualities proves that the explanation is the process that produced the result.

This distinction matters because an explanation can do several jobs. It can help a user understand a decision, help an engineer debug a model, justify an action to an auditor, expose a sensitivity, or merely make an output feel less arbitrary. A sentence that succeeds at one job may fail at another. The mistake is treating every useful story as a window into mechanism.

Separate four objects that ordinary language calls an explanation

First is the output: the label, answer, score or action. Second is the evidence available to a reviewer: inputs, logs, retrieved documents, model cards, tests and known limitations. Third is an explanatory representation: a feature attribution, example, rule, counterfactual or natural-language rationale. Fourth is the actual computational process that transformed system state and input into output.

These objects can overlap, but they are not identical. A simple decision tree may expose much of its operative rule. A feature-attribution method may instead approximate which input changes would alter one prediction. A language model may generate a retrospective rationale because the interface asked for one. The rationale can be coherent even when it is not a faithful trace of the hidden computation.

The point is not that all explanations are fake. It is that the word explanation does not specify the relationship being claimed. Before trusting one, ask what object it represents and what evidence supports that representation.

NIST separates intelligibility from accuracy

The U.S. National Institute of Standards and Technology proposed four principles for explainable AI: a system supplies an explanation; the explanation is meaningful to its intended recipient; it accurately reflects the reason or process; and the system recognizes its knowledge limits. The separation is deliberate. An explanation can be understandable without being accurate, and accurate without being useful to a particular audience.

NIST also treats transparency, explainability and interpretability as related but distinct trustworthiness characteristics. That framing resists a common shortcut: adding a friendly paragraph to an opaque workflow does not automatically make the system transparent, valid, reliable or accountable. Those properties require different evidence.

Sourced fact: NIST's framework distinguishes explanation accuracy from meaningfulness and places explainability beside, not above, validity, reliability, safety, privacy and accountability. Inference: therefore an organization should not use the presence of an explanation as a substitute for measuring whether the system works for the decision at hand.

Plausibility is a human judgment; faithfulness is a claim about the system

A plausible explanation fits what a person expects. A faithful explanation tracks the model's operative behavior closely enough for the stated purpose. These can diverge. A heat map can highlight the part of an image that a human considers important while remaining insensitive to trained model parameters. A natural-language rationale can mention sensible evidence while the output changes for an unrelated token or formatting choice.

Adebayo and colleagues proposed sanity checks for saliency maps and found that some methods could produce visually convincing results even when model parameters or training labels were randomized. Later researchers have challenged parts of the methodology and emphasized how difficult ground-truth evaluation remains. That disagreement is useful. It shows that a compelling visualization is not self-validating and that the test must match the explanation claim.

If a vendor says a highlighted phrase caused a classification, ask whether removing or changing that phrase predictably changes the output. If it says an explanation describes the internal process, ask how faithfulness was measured. If it says the explanation helps reviewers detect mistakes, ask for reviewer-performance evidence rather than a gallery of attractive examples.

Counterfactuals can guide action without revealing a mechanism

Suppose a lending system says an application would cross a threshold if reported debt were lower. That counterfactual may help a reviewer understand the local decision boundary. It does not necessarily show how every internal component represented debt, which correlated variables mattered, or whether the same change would produce the expected result after the system is updated.

Usefulness is still real. A map does not need to reproduce every molecule in a landscape to help someone choose a road. But a road map should not be sold as geology. The appropriate claim is narrow: this representation shows a tested relation between specified changes and this version of the system under stated conditions.

A model's own rationale needs a second source of evidence

When a generative model explains its own answer in ordinary language, the explanation is another output. It may summarize retrieved evidence, reconstruct a defensible route, or produce a socially expected justification. Without independent instrumentation or controlled tests, the text alone cannot establish which internal computation was decisive.

This does not make chain-of-thought-style prompting worthless. Asking for assumptions, sources, intermediate results or uncertainty can improve a user's ability to inspect an answer. The practical value lies in the inspectable artifacts, not in treating private verbalized reasoning as a certified transcript of cognition.

Prefer requests such as “show the source passage,” “state which premise would change the conclusion,” “calculate the result in a second way,” and “name the boundary of your evidence.” These produce claims that can be checked. A long autobiographical story about how the model thought can merely increase the surface area of confidence.

Researchers disagree about what to build for high-stakes use

Cynthia Rudin argues that high-stakes decisions should often use inherently interpretable models instead of applying post-hoc explanations to black boxes. The argument is not that simple models always win. It is that an approximate explanation introduces another model whose errors and incentives must be governed, especially when interpretable alternatives can achieve adequate performance.

Other work continues to improve local explanations, feature attribution, mechanistic interpretability and human-centered interfaces. The field has not converged on one method or one definition. Different systems and users need different forms of understanding. A safety engineer, affected applicant and product operator are not asking the same question.

Expert disagreement: there is active debate about when post-hoc explanation is sufficient and when only an interpretable-by-design system is defensible. Stable conclusion: the burden should rise with consequence, and explanation claims need validation rather than aesthetic confidence.

Use a seven-question explanation audit

1. Decision: What exact output or system behavior is being explained?
2. Audience: Who needs the explanation, and what action must it support?
3. Type: Is this a rule, example, attribution, counterfactual, trace, summary or generated rationale?
4. Scope: Does it explain one output, typical behavior or the whole system?
5. Faithfulness: What perturbation, comparison or ground truth tests the claimed relation?
6. Stability: Does the explanation survive harmless changes, reruns and model updates?
7. Consequence: What independent evidence, appeal and human authority remain when it is wrong?

The audit does not require every user to become a machine-learning researcher. It requires the seller or operator to name the kind of transparency being offered. “Here are the sources the answer cited” is useful. “Here is why the system decided” is a stronger claim.

Test explanations in the deployment, not only in the demonstration

Explanations change behavior. They can help people notice errors, but they can also automate trust by making a prediction feel considered. Evaluate whether reviewers catch planted failures, whether different audiences interpret the same display consistently, and whether the explanation encourages appropriate override rather than ritual approval.

Pair this with the local-validity work in An AI Benchmark Is Not the Real World. Measure the underlying task outcome, the explanation's faithfulness for its declared scope, and the human decision made with it. Then preserve the responsibility chain described in Automation Does Not Remove Responsibility. A reason box cannot own harm or repair it.

The bottom line

An explanation is a designed representation. It may be accurate, approximate, useful, misleading or several of those at once for different audiences. The honest question is not whether the system has explanations. It is which relationship the explanation claims to represent, how that relationship was tested and what happens when the representation fails.

Use explanations to create inspectable questions. Do not let them close the inquiry merely because they sound like answers.

Research and standards notes

• NIST, Four Principles of Explainable Artificial Intelligence.
• NIST AI Resource Center, AI risks and trustworthiness characteristics.
• Adebayo and colleagues, Sanity Checks for Saliency Maps.
• Rudin, Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead.
• For answer-level checking, use How to Verify an AI Citation Before You Use It and A Model Can Be Right for the Wrong Reason.