See how group weights reverse an algorithm error-rate comparison, standardise to a common regime mix, and separate arithmetic from causal adjustment.
Simpson paradox
Simpson paradox is a reversal between an association measured in every subgroup and the association measured after those groups are pooled. In quant work, it can make an algorithm with a lower error rate in every market regime look worse overall. The missing piece is often the mix of observations each algorithm received. Pearl’s account distinguishes this arithmetic reversal from the causal conclusions people are tempted to draw from it.
The algorithm that loses every comparison—and wins the total
Consider a fictional evaluation of two algorithms, A and B. Each observation is one prediction, classified as either an error or a non-error, and assigned to exactly one of two regimes. Lower error is better. These counts illustrate the arithmetic; they are not market evidence.
Regime
A: errors / predictions
A: error rate
B: errors / predictions
B: error rate
Calm
8 / 80
10%
1 / 20
5%
Volatile
12 / 20
60%
40 / 80
50%
Pooled
20 / 100
20%
41 / 100
41%
B has the lower conditional error rate in each regime: 5 percentage points lower in calm conditions and 10 percentage points lower in volatile conditions. Yet A’s pooled error rate is 21 percentage points lower than B’s.
There is no inconsistent addition. A received 80 calm predictions and only 20 volatile predictions. B received the opposite mix. Volatile predictions have a much higher error rate for both algorithms, so B’s pooled result puts much more weight on the harder regime. Having 100 predictions per algorithm does not make the regime composition comparable.
Fictional samples: A has 80 calm and 20 volatile predictions; B has 20 calm and 80 volatile. Calm error rates are A 10% and B 5%; volatile rates are A 60% and B 50%. Actual pooled rates are A 20% and B 41%. Common 50:50 weighting gives A 35% and B 27.5%.
Original fictional evaluation. The upper panel shows the different denominators; the lower panel uses a common error-rate axis. Standardised rates are weighted averages, not observed counts from another sample.
Put the weights where you can see them
Let E mean an error, M the algorithm and G the regime. The law of total probability, applied separately within each algorithm’s sample, gives
P(E∣M=m)=g∑P(E∣M=m,G=g)P(G=g∣M=m).
The first term is the regime-specific error rate. The second is that regime’s share of the algorithm’s predictions. Substituting the table,
A, pooled:B, pooled:0.8(0.10)+0.2(0.60)=0.20,0.2(0.05)+0.8(0.50)=0.41.
The two averages use different weights. Comparing them combines differences in error rates with differences in exposure.
For a descriptive comparison at a common 50:50 regime mix, use the same weights for both algorithms:
A, standardised:B, standardised:0.5(0.10)+0.5(0.60)=0.35,0.5(0.05)+0.5(0.50)=0.275.
B’s standardised rate is 27.5%, versus A’s 35%: a difference of 7.5 percentage points in B’s favour. The 50:50 mix is an explicitly chosen comparison population, not an estimate of tomorrow’s regime frequencies.
More generally, if a common calm weight is q∈[0,1], A’s weighted rate minus B’s is
q(0.10−0.05)+(1−q)(0.60−0.50)=0.10−0.05q>0.
So B has the lower rate under every common regime mix in this example. The observed reversal arises because the actual mixes differ. The Conditional Expectation lesson develops the same weighting principle through total expectation; its full teaching requires membership, with a public preview available.
Which comparison answers your question?
The pooled rates describe errors in the samples actually observed. Common-weight rates describe a specified shared mix, assuming the displayed conditional rates apply there. Neither association is automatically a causal effect of switching algorithms.
For a causal question, choose adjustment using the assumed data-generating graph and the effect you want to estimate. If regime is a sufficient pre-assignment confounder, appropriate adjustment may identify that effect under consistency, overlap and exchangeability assumptions. If a grouping variable is a mediator, conditioning changes the target away from the total effect; if it is a collider, conditioning can introduce bias. A reversal alone cannot choose between these stories. See Pearl, §2.2–2.3 and the backdoor criterion.
For this fictional table, report the counts, conditional rates, actual weights and chosen comparison mix together. There is no assignment mechanism or uncertainty model here to establish causality or statistical significance. Picking whichever aggregation produces the preferred winner would answer neither question reliably.
Sources
E. H. Simpson (1951), The Interpretation of Interaction in Contingency Tables, especially §§8–11, pp. 240–241. The card and treatment examples show why the interpretation of a combined table depends on the story behind its categories.
Judea Pearl (2014), Understanding Simpson’s Paradox, UCLA author manuscript R-414; §1 and §§2.1–2.3. Separates association reversal, causal interpretation and graphical adjustment.
