Reverse engineer the computations behind a model behaviour, using ablation and activation patching while distinguishing internal influence, circuit completeness and economic causality.
Mechanistic interpretability
Mechanistic interpretability is the attempt to explain a trained model's behaviour by reverse engineering the internal computations that produce it. For a quant investigating a financial-text classifier, this means asking how representations and connections produce a selected score—for example, whether a suspected shortcut actually influences the prediction. It is a branch of explainable AI whose explanations are tested against the model's machinery.
A circuit is a proposed collection of internal features or components and their connections that implements a particular behaviour. The explanation might involve neurons, directions in activation space, attention heads, or paths between layers. Its scope is a named behaviour in a particular model, rather than a complete account of everything that model can do.
Watching a signal is the start of the investigation
Suppose a hidden activation rises when an earnings report contains negative language. A probe trained to recognise negative language from that activation shows that the information is recoverable there. It does not establish that the model uses that information to produce its final score. Likewise, a striking attention pattern is a candidate mechanism to investigate, not a finished causal explanation.
Mechanistic work connects observations to an algorithm and then tests its predictions. In the original Transformer Circuits framework, induction heads help implement a copying pattern: after seeing an earlier sequence A followed by B, a later A promotes B as a continuation. The account connects where a head attends with what it writes toward the output. Those initial results concern small, simplified attention-only transformers; they do not amount to a complete explanation of a large language model.
Change one internal quantity
Consider an entirely specified toy circuit with dimensionless activations a and b:
y=a+b.
In the original run, a=3 and b=2, so the output is 5. Zero ablation overwrites the selected activation with zero. Set a to 0, keep b=2, and evaluate the downstream sum again:
yoriginal=3+2=5,yablated=0+2=2,Δy=5−2=3.
This intervention establishes an influence of three output units inside this stipulated computation, under this overwrite and fixed value of b. The activations have no economic labels. It says nothing about whether changing a firm's language, leverage or earnings would cause a three-unit change in a real financial outcome. That would require a separate causal inference argument about the world.
Original toy circuit: a equals 3 and b equals 2 give output 5. Zeroing a while holding b at 2 gives output 2, a drop of 3. Restoring cached a equals 3 returns output to 5. All quantities are dimensionless.
Original stipulated circuit. The three runs isolate an internal overwrite and its reversal; they are not observations of a trained financial model.
Zero is an intervention choice, not a universally neutral state. A real network may never naturally combine a zeroed component with the other surviving activations. A large effect could therefore include the network's response to an unusual internal state. Mean replacement and activations drawn from other examples ask different questions; none automatically preserves the natural joint distribution of activations.
Put a recorded activation back
Activation patching replaces a chosen internal activation in one run with a value recorded in another. A common experiment records a clean run, constructs a contrasting or corrupted input, then restores selected clean activations during that second computation. The downstream calculation is rerun, and recovery is measured against the corrupted run. Meng and colleagues' causal tracing uses corruption and restoration to investigate factual recall in language models.
The last row of the toy figure is a simple restoration: copying the original a=3 back into the ablated run returns the output to 5. In a network, recovery supports a claim about the patched site under the chosen contrast. It does not identify every computation upstream or prove that the activation has one exclusive semantic role.
The experiment's design is part of its meaning. Zhang and Nanda find that changing corruption methods and evaluation metrics can change which components appear important. Record the exact input pairs, replacement rule, layer and position, components allowed to recompute, and output metric. A change in a target token's probability can tell a different story from a change in its logit relative to a competing token. Test multiple relevant inputs and plausible interventions before generalising beyond the examples examined.
Influence, necessity and sufficiency ask different questions
An ablation can test necessity for a specified performance criterion: does removing a component prevent the behaviour under the stated conditions? The toy's a is needed to preserve the exact output 5 when b=2; it is not needed to produce a positive output, because 2 remains. A measured output difference alone does not define the criterion.
A sufficiency test asks whether a proposed circuit can retain or restore the behaviour when other routes are disabled or controlled in a specified way. Restoration in one damaged run is conditional evidence, rather than proof that the component alone can implement the task. Neither test establishes uniqueness: alternative paths may implement similar computations or compensate when another path is removed.
Wang and colleagues' indirect-object-identification study found backup attention heads that became influential after other heads were ablated. Their circuit evaluation examined fidelity to the model, omitted task-relevant structure and unnecessary components, and still exposed gaps. A plausible circuit diagram can therefore be useful and incomplete at the same time.
For a financial model, the useful deliverable is a testable account of a specific computation: the model version, behaviour, input population, proposed paths, interventions and failures of the account. Understanding that computation can help debugging. It does not validate the forecast, establish market causality, or guarantee that the same mechanism will dominate in another regime.
