An ablation study compares a full system with specified components removed or changed. Learn retraining versus masking, interaction effects and fair evaluation.
Ablation study
An ablation study compares a complete system with versions in which specified components are removed, replaced or disabled. In quant research, it helps answer whether a feature family, preprocessing step or model component earns its complexity under a stated evaluation protocol. A strong full-model result alone cannot tell you which additions helped.
The intervention must be explicit. Retraining without a component asks what the learning procedure achieves without it. Disabling a component in an already fitted system asks how that fixed system behaves when the component is unavailable. Both are used in ablation research, but they answer different questions: Durrani, Dalvi and Sajjad, §§4.1–4.2, distinguish test-time activation masking from classifier retraining on selected neurons. Here, every variant is retrained from scratch. Zeroing an input in the already fitted full model would be a different experiment and can present it with an unfamiliar input pattern.
First decide what is being compared
Suppose a classifier predicts a future market event using two feature families, A and B, on top of a common baseline. A family might contain several transformations of the same information source. Removing a family together can be more informative than removing one column that has close substitutes.
Keep the target, eligible observations, training and held-out boundaries, scoring rule and evaluation code common across variants. Declare what replaces the removed component and how training changes. For example, removing an input family changes the input dimension; it must not also change which observations survive missing-data filtering without disclosure.
The comparison should also expose the tuning budget. Holding settings fixed asks about removal under that configuration. Retuning each variant under the same declared search budget asks about achievable performance under that budget. Giving the full model much more tuning effort confounds the comparison. Lipton and Steinhardt discuss how gains attributed to architectural innovations can instead come from tuning.
A concrete research example appears in Attention Is All You Need, §6.2 and Table 3: the authors vary Transformer components and measure development-set performance, including an attention-head comparison that holds computation constant. That is an example of specifying the comparison; it does not imply one universal resource-control rule for every ablation.
Four scores reveal an interaction
Consider these fictional held-out development scores, stipulated for four separately retrained variants. All use the same samples and protocol. The metric is unadjusted balanced accuracy: for a binary classifier, the mean of the recall for the two classes, as defined in the official metric documentation. These numbers are not measured market results.
Variant
Included feature families
Balanced accuracy
Baseline
Neither A nor B
0.50
A only
A
0.54
B only
B
0.56
Full model
A and B
0.65
Let S0, SA, SB and SAB denote those scores. Removing A from the full procedure leaves B only, so the drop is 0.65−0.56=0.09, or 9 percentage points. Removing B leaves A only, giving 0.65−0.54=0.11, or 11 percentage points.
Yet adding A to the baseline earns only 0.54−0.50=0.04, or 4 percentage points; adding B alone earns 6 percentage points. The removal and standalone effects differ because the contribution depends on the companion family.
An interaction contrast on this particular score scale makes the difference explicit:
I=SAB−SA−SB+S0=0.65−0.54−0.56+0.50=0.05.
The full gain of 15 percentage points therefore decomposes into 4+6+5. The extra 5 points describe nonadditivity of these stipulated scores. They are not a uniquely allocated contribution to A or B, a significance test or proof that the two information sources interact through a causal market mechanism. Changing the metric can also change this contrast.
Fictional balanced accuracy is 50% for baseline, 54% for A only, 56% for B only and 65% for A plus B. Removing A loses 9 percentage points; removing B loses 11. Standalone gains are 4 and 6 points, with a positive 5-point interaction on this score scale.
Original fictional example. All variants are retrained on the same protocol; the scores illustrate conditional contributions rather than empirical evidence.
A near-zero removal effect would not prove a family contains no information: another family may substitute for it, or the chosen learner may fail to use it. An improvement after removal would instead be evidence worth investigating about complexity, noise or training behaviour under that protocol.
What makes the result useful for a trading model?
An ablation is a comparison within model validation, rather than a substitute for it. Refit learned preprocessing and any feature selection inside each training fold. The official leakage guidance explains why fitting transformations on held-out data invalidates performance estimates. For a historical trading replay, also enforce information availability and remove training labels whose outcome intervals overlap the evaluation interval where required; common leakage across every variant still invalidates the comparison.
Pair comparisons on the same splits and examine repeated training seeds and different eligible time blocks. A fixed seed aids reproduction; one seed does not establish stability. Report the spread of paired differences and use an uncertainty method suited to temporal dependence rather than pretending repeated scores are independent. The four illustrative scores above provide no sample size, prediction records or repeated runs, so they establish no empirical significance.
If ablation results guide which system to retain, they are development evidence. Freeze that selection procedure before using an untouched final test. Time-Ordered Train, Validation and Test Design develops the information boundaries and refitting schedule; its preview is public and the full lesson requires membership.
Finally, classification quality and strategy economics are different objects. Compare realistic transaction costs, turnover, sizing and risk alongside predictive metrics. A family can improve balanced accuracy while concentrating its correct predictions in trades too small or costly to exploit. The useful conclusion states which component, which intervention, which baseline, which protocol and which metric supported the decision.
Sources
Lipton, Z. C. and Steinhardt, J. (2018), Troubling Trends in Machine Learning Scholarship, §§3.2 and 5.1. Identifying the source of gains, tuning confounds and the limits of ablation as an explanation.
Vaswani et al. (2017), Attention Is All You Need, §6.2 and Table 3. Primary research example of controlled component comparisons on development data.
Durrani, N., Dalvi, F. and Sajjad, H. (2023), Discovering Salient Neurons in deep NLP models, §§4.1–4.2, pp. 10–11. Test-time activation masking and retraining on a selected neuron subset are evaluated separately.
scikit-learn developers, Common pitfalls and recommended practices, §§12.2–12.3. Training-only learned transformations, comparable splits and random-state behaviour.
scikit-learn developers, balanced_accuracy_score, definition and adjusted parameter. Official definition used for the fictional score scale.
