Skip to content

Generalisation

How a fitted model performs on new data, why perfect training fit is insufficient, and how deployment targets and chronological evaluation define a credible test.

Generalisation is a fitted model's ability to perform on new observations beyond those used to develop it. For a return forecaster, the relevant question is whether predictions remain useful when new market outcomes arrive. Reproducing historical labels perfectly is a different achievement.

The claim needs a destination: new observations from which assets, at which forecast horizon, under which information and market conditions? A model can generalise to further observations from a familiar population yet fail when the traded universe or market mechanism changes.

The error you fit and the error you need

Let f^\hat f be a model after fitting, XX its inputs and YY its prediction target. With squared-error loss, its generalisation error under a specified deployment distribution QQ is

RQ(f^)=E(X,Y)∼Q[(Y−f^(X))2].R_Q(\hat f)=\mathbb{E}_{(X,Y)\sim Q}\left[(Y-\hat f(X))^2\right].
This expected value concerns new observations, holding the fitted model fixed. Averaging over possible training samples would answer an additional question about the learning procedure. Other tasks can use other losses: a default classifier, for example, may need probability calibration and asymmetric decision costs rather than squared return errors.
By contrast, training mean squared error averages losses on the observations used to fit the model. A held-out test average estimates new-data performance only to the extent that its sampling and evaluation design represent the intended use. The introductory training/test distinction is developed in ISLR §2.2.1.

Classical analyses often assume independent training and test observations drawn from the same distribution. Forecasting future returns requires more care: neighbouring observations can be dependent, and the future distribution can differ from the historical one. A low estimated error under one distribution does not establish the same error under another.

Perfect memory, imperfect forecasts

Consider a fictional model that achieves training MSE of zero. It then makes two frozen forecasts of next-session simple returns. Features are available when each forecast is issued; its return label becomes observable only after the forecast session finishes. Returns are quoted in basis points, so squared errors have units of basis points squared.

Unseen observationForecastRealised returnRealised minus forecastSquared error
A1211
B31−24
Test MSE=(2−1)2+(1−3)22=2.5 bps2.\text{Test MSE}=\frac{(2-1)^2+(1-3)^2}{2}=2.5\ \text{bps}^2.
Two synthetic next-session return forecasts: observation A predicts 1 basis point and realises 2; B predicts 3 and realises 1. Squared errors are 1 and 4 basis points squared, averaging 2.5. Perfect training fit does not remove these unseen errors.
Two synthetic next-session return forecasts: observation A predicts 1 basis point and realises 2; B predicts 3 and realises 1. Squared errors are 1 and 4 basis points squared, averaging 2.5. Perfect training fit does not remove these unseen errors.
Original fixed synthetic example, with returns in basis points and squared errors in basis points squared. Both forecasts are frozen before their respective outcomes become observable; the two rows are not a reliability study.
The arithmetic disproves the inference “training error is zero, therefore future error is zero”. It does not show that this model is reliably good or bad: two observations give little evidence about the range of errors, dependence or rare outcomes in deployment. Nor does an error of 2.5 have an intrinsic pass/fail meaning. Compare it with a declared baseline and the economic purpose of the forecast.
A small training-to-test gap also need not mean useful prediction. Both errors can be large. Conversely, fitting the training data exactly does not by itself prove overfitting; the concern is whether development choices capture sample-specific patterns that fail to transfer.

Make “new” an enforceable boundary

For an equity return model, define the eligible universe, prediction timestamp, horizon, target construction and planned update schedule before interpreting its score. Then test the full procedure that could actually run:

  • Fit with information available then. Completed labels and point-in-time features must precede their use. Otherwise look-ahead bias can make an apparent success depend on unavailable information.
  • Keep selection separate from assessment. Use development data to choose features, tuning settings and preprocessing. Once a final test set influences those choices, it has joined development. Official scikit-learn guidance explains why preprocessing also belongs inside the training boundary.
  • Match the split to the claim. Chronological evaluation addresses future-time forecasting; holding out entire issuers addresses transfer to unfamiliar issuers. Randomly mixing dependent rows can make the test artificially similar to training. The official time-series evaluation discussion explains this limitation.
  • Assess uncertainty and market change. Report performance across suitable time blocks and relevant conditions, with uncertainty methods that respect dependence. Monitor genuinely arriving observations after development; more historical rows do not guarantee coverage of a new market regime.
Time-Ordered Train, Validation and Test Design turns these boundaries into a refitting schedule and handles unfinished labels. The full lesson requires membership; a public preview is available.

Generalisation of a forecast is one component of a trading decision. Even a credible predictive improvement can be consumed by turnover, fees, slippage or capacity limits. Evaluate the forecast's loss and the strategy's economic outcome as distinct claims.

Sources

Last updated
  • machine learning
  • prediction
  • evaluation
  • financial data
Keep connecting the ideas
← Back to the glossaryBack to top ↑