Expected calibration error: Definition & Examples

Expected calibration error (ECE) summarises mismatches between predicted probabilities and observed event frequencies by averaging absolute gaps across probability bins, weighted by the number of predictions in each bin. For a quant model feeding probabilities into position sizing, it asks whether the numbers behave like frequencies. A small ECE alone cannot establish that the model is useful or safe to size from.

First name the probability being checked

There are different ECE conventions. This article's worked example checks binary positive-class probabilities:

p_i

estimates

\mathbb{P}(Y_i=1\mid X_i)

, and

y_i

is the realised zero-or-one outcome. Probability calibration means that among forecasts near

p

, the positive event occurs with frequency near

p

. This is the positive-instance formulation in Naeini et al. (2015).

The widely used top-label ECE in Guo et al. (2017) instead bins the probability assigned to the predicted class and compares it with classification accuracy. In binary classification, that confidence is

c_i=\max(p_i,1-p_i)

; the outcome being averaged becomes

\mathbf{1}_{\{\hat y_i=y_i\}}

. For

p_i=0.2

, the positive-class probability is

0.2

, while confidence in the predicted class 0 is

0.8

. State the class-selection and tie rules when calculating top-label ECE. These two inputs and outcomes must not be silently interchanged. In multiclass problems, top-label ECE also omits the other class probabilities.

A weighted gap, not a classification error rate

For nonempty bins $B_m$ containing $n_m$ of the $n$ observations, define

\bar p_m=\frac{1}{n_m}\sum_{i\in B_m}p_i, \qquad \bar y_m=\frac{1}{n_m}\sum_{i\in B_m}y_i.

The empirical positive-class score is

\widehat{\mathrm{ECE}}_{+} =\sum_{m:n_m>0}\frac{n_m}{n}\lvert\bar y_m-\bar p_m\rvert.

An empty bin contributes zero. The absolute value is taken after averaging within each bin. Averaging

\lvert y_i-p_i\rvert

observation by observation measures something different. This weighted-bin construction follows Naeini et al., “Calibration Measures”.

Consider ten fictional forecasts of a positive financial event, with a fixed event definition and horizon:

Observation	1	2	3	4	5	6	7	8	9	10
Predicted $p_i$	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Realised $y_i$	0	0	1	0	1	0	1	1	0	1

Use bins $[0,0.5)$ and $[0.5,1]$ , explicitly including 1 in the upper bin.

Bin	Count	Mean probability	Positive frequency	Weighted absolute gap
$[0,0.5)$	4	0.25	$1/4=0.25$	0
$[0.5,1]$	6	0.75	$4/6=2/3$	0.05

Hence

\widehat{\mathrm{ECE}}_{+} =\frac{4}{10}|0.25-0.25| +\frac{6}{10}\left|\frac{2}{3}-0.75\right| =0.05.

That is a five-percentage-point weighted calibration gap, not “5% of labels predicted incorrectly,” and not a claim that every forecast is within five points of its true probability.

![Fictional positive-class calibration: bins 0, 0.5) and [0.5, 1] contain 4 and 6 predictions, with reliability points (0.25, 0.25) and (0.75, 0.6667). ECE is 0.05; assigning p equal to 0.5 to the lower bin changes ECE to 0.15.

Original fictional example. The vertical distance from each point to the diagonal is its bin gap; ECE weights those gaps by bin counts. Changing the endpoint rule changes the summary without changing a forecast or outcome.

Move only the endpoint convention to $[0,0.5]$ and $(0.5,1]$ . The forecast at $0.5$ moves into the lower bin. Counts become 5 and 5; mean probabilities become $0.3$ and $0.8$ ; positive frequencies become $0.4$ and $0.6$ . The score is now

\frac{5}{10}|0.4-0.3|+\frac{5}{10}|0.6-0.8|=0.15.

The model has not changed. The reported ECE has tripled because the bin definition changed. These are intentionally tiny teaching data, not evidence of model quality.

INTERACTIVE EXPLORER

Binary probability calibration

Change probability bins and inspect the weighted calibration gap.

What the scalar can hide

Cancellation inside a bin. Consider a separate fictional sample with two equally sized groups: one predicts

0.2

and has positive frequency

0.4

; the other predicts

0.8

and has positive frequency

0.6

. Pool them into one bin and both averages are

0.5

, giving ECE zero. Separate the groups and ECE is

0.2

. Underprediction and overprediction cancelled before the absolute value. Nixon et al., Sections 3.3–3.4 discuss this masking and the tradeoff between finer bins and noisier bin estimates.

Estimation is not population truth. The population quantity for this binary variant is

\mathbb{E}[|\mathbb{E}[Y\mid P]-P|]

, where

P

is the model's probability forecast. A finite binned estimate adds sampling noise and aggregation effects. Holding a coarse partition fixed need not recover that quantity even with many observations; finite-sample ECE is not guaranteed to be a lower bound. Vaicenavicius et al., Sections 4.1–4.3 distinguish these asymptotic and sampling issues.

Calibration is not all predictive quality. A constant forecast equal to the population event rate can be calibrated while ignoring useful information in

X

. ECE is a calibration diagnostic, not a proper scoring rule for individual forecasts. It cannot replace a predictive loss that rewards truthful conditional probabilities. Report Brier score or log loss alongside calibration, ranking and decision-relevant evaluation. Proper-loss training itself does not guarantee calibration on new data; see Nixon et al., Section 2.2.

Use it as an audit trail

Report the variant, event label and horizon, evaluation sample size and period, bin edges and endpoint rules, bin counts, and each bin's signed gap. A calibration curve shows where the scalar comes from. Keep binning conventions consistent when comparing models and assess sensitivity to reasonable alternatives.

Fit any recalibration map on training or calibration data, then evaluate it on separate held-out outcomes. Repeatedly choosing a calibrator or bins to minimise the final test ECE makes that test part of model selection. For financial time series, respect chronology and overlapping label windows; an overall score can also conceal failures in important regimes. The owned Probability calibration lesson connects these validation choices to sizing inputs; its full teaching requires membership, with a public preview.

Sources

Naeini, Cooper and Hauskrecht (2015), Obtaining Well Calibrated Probabilities Using Bayesian Binning, “Calibration Measures,” p. 2903: positive-instance frequencies, mean probabilities and bin weights.
Guo et al. (2017), On Calibration of Modern Neural Networks, Section 2, equations 1–6: predicted-class confidence calibration, binned ECE and negative log likelihood.
Nixon et al. (2020 revision), Measuring Calibration in Deep Learning, Sections 2.1–2.2 and 3.1–3.4: metric conventions, omitted class probabilities, proper losses and binning pathologies.
Vaicenavicius et al. (2019), Evaluating model calibration in classification, Sections 4.1–4.3: binned estimation, asymptotic bias and estimator variability.