Elicitability is the property of a statistical target being the unique minimiser of an expected scoring function, over a specified class of distributions. It gives a quant a principled way to compare forecasts: choose a score that rewards the statistic actually requested, then evaluate forecasts against later outcomes. A score aimed at a mean can mis-rank forecasts intended to estimate a tail quantile.
Let be the target extracted from distribution , a reported value and the realised loss. With lower scores preferred, strict consistency means
The same function must work for every in the declared class, with the required expectations well defined. It cannot be redesigned using the unknown true target for each distribution. Elicitability says that such a score exists; it does not identify one universally best score. See Gneiting, §§2.2–2.3.
Asking a score for a quantile
Value at risk at confidence is a loss quantile. For a loss distribution with a finite first moment and a unique -quantile, the pinball score is strictly consistent:
Under-reporting a realised loss costs ; over-reporting it costs . The asymmetry asks for the chosen quantile rather than the mean. When quantiles are not unique, this score can have several minimisers; strict uniqueness cannot be assumed just because software returns one number.
Consider fictional one-day losses in USD:
| Loss | Probability | Cumulative probability |
|---|---|---|
| 0 | 50% | 50% |
| 1 | 30% | 80% |
| 4 | 20% | 100% |
At , the unique quantile is USD 1: the cumulative probability jumps across 75% there. Expected scores are probability-weighted averages:
| Report (USD) | Expected pinball loss (USD) |
|---|---|
| 0 | 0.825 |
| 1 | 0.575 |
| 4 | 0.725 |
For the truthful report,
This is the minimum over all real reports, not just the three rows. The expected score has slope below zero, between zero and one, between one and four, and above four. It decreases up to one and increases afterwards.
Original fictional example. The line shows expected score as the report changes, not a realised loss path. The ES report is truthful for a different target, so it need not minimise a quantile score.
For the weighted-average step, Expectation, Variance and Covariance develops expectation from discrete probabilities. Its public page is a preview; the full lesson requires a free signed-in account.
ES alone and ES alongside VaR
Expected shortfall averages the worst specified probability mass. Here the worst 25% contains all 20% at USD 4 and five percentage points of the mass at USD 1:
The strict-exceedance average selects only 20%, so it is a different statistic. Evaluating the ES report 3.4 with pinball loss gives 0.695, above the score at 1. That demonstrates a target mismatch; it does not prove that ES lacks every possible standalone score.
The impossibility result concerns ES as a single report on sufficiently rich distribution classes. Gneiting's Theorem 11 covers classes containing all finitely supported distributions on an interval. It is not a claim about every restricted family: on deterministic-loss distributions, for example, ES equals the loss itself and squared error elicits it.
However, the pair VaR and ES is jointly elicitable. Under finite first moments and unique relevant quantiles, suitable scores evaluate the two reports together, with upper-tail loss reports satisfying . The expected joint score is uniquely minimised at the true pair ; it cannot in general be split into an independent VaR score plus an independent ES score. Fissler and Ziegel, Corollary 5.5, prove this using lower-tail notation; reversing the outcome sign gives the upper-tail convention here. Their 2019 correction, §2, preserves the result on the maximal sensible VaR/ES action domain.
What the property lets you conclude
For forecast comparison, use the same target, horizon, outcome convention, observations and chosen consistent score for competing methods. Averaging realised scores estimates an expected comparison; finite samples, tail scarcity and serial dependence still affect uncertainty. Different consistent scores can rank imperfect forecasts differently.
Calibration asks a different question: whether forecasts satisfy the relevant distributional restrictions, possibly conditional on available information. Winning a score comparison does not establish calibration, and ES's standalone non-elicitability does not make ES backtesting impossible. Fissler, Ziegel and Gneiting, pp. 2–3, distinguish model verification from comparative backtesting.
Elicitability also says nothing by itself about easy estimation, next-period accuracy or subadditivity. Aggregating portfolio risk and scoring a forecast are separate mathematical tasks.
Sources
- Tilmann Gneiting (2011), Making and Evaluating Point Forecasts, §§2.2–2.3, §3.3 and §3.4, especially Theorem 11: strict consistency, quantile scores and the distribution-class scope of standalone ES non-elicitability.
- Tobias Fissler and Johanna F. Ziegel (2016), Higher order elicitability and Osband's principle, Corollary 5.5 and Remark 5.3: joint VaR/ES scores. Read with their 2019 correction and supplement, §2, particularly the maximal action-domain case on p. 10.
- Tobias Fissler, Johanna F. Ziegel and Tilmann Gneiting (2016), Expected Shortfall is jointly elicitable with Value at Risk—Implications for backtesting, pp. 1–3: joint scoring, model verification and forecast comparison. The linked author preprint is dated July 2015; the numerical example and figure here are original.