A validation set helps choose a model and its settings. Understand its role in chronological financial research, preprocessing and repeated model searches.
Validation set
A validation set is data used to compare candidate models or settings after those candidates have been fitted on other data. In quant research, it helps decide which forecasting specification, signal threshold or tuning setting to carry forward. Because its results influence that choice, validation belongs to development: its winning score is not an untouched estimate of final performance.
Take 100 synthetic observations ordered by decision time. Assign rows 1–60 to training, 61–80 to validation and 81–100 to the final test. The validation block contains 20 observations; training and validation together supply 80 development observations. This allocation is illustrative, not a universal split ratio.
Synthetic chronological assignment: rows 1–60 are training, 61–80 validation and 81–100 final test. Training fits candidates, validation chooses settings and can prompt revisions, and the final test assesses the frozen procedure.
Original synthetic split. The upper block widths represent row counts; the lower panels show information use. Allocation alone does not establish that a row’s label was available for fitting.
Fit each candidate on eligible training rows, then compare its predictions on the validation block using a declared metric. Validation might favour a different regularisation strength, a simpler model or retaining the baseline. A predictive score and a strategy’s economic performance answer different questions: if the selection criterion is trading performance, specify costs, execution assumptions and sizing before the comparison.
Once selection is complete, a predeclared final refit may use eligible training and validation data before final testing. That is a later fitting step. It does not mean validation data were permitted in the original candidate fits, and it does not restore the validation score’s independence. Keep the final-test data outside both fitting and selection.
Financial labels need an information clock
For a future-return target, specify the decision time, forecast horizon and when the outcome becomes observable. A row can precede the validation block while its label finishes inside it. Such a target is unavailable at the earlier fit cutoff; chronological row allocation alone does not make it eligible. Likewise, construct features from data actually available at each historical decision, including publication delays and revisions.
For the candidate comparison, fit data-dependent preprocessing on the eligible training rows and apply those fitted transformations to validation. Estimating a scaler, imputation rule or feature selection from the combined training and validation sample lets held-out information affect the fitted pipeline. The scikit-learn leakage guide explains why preprocessing belongs inside the fitting boundary.
The owned lesson Time-Ordered Train, Validation and Test Design develops label eligibility, preprocessing and scheduled refitting in a historical replay. Its full text requires membership; the opening and outline have a public preview.
Reusing validation spends its independence
Validation supports choices, so some adaptation to it is intentional. The problem is treating the best result after a large search as an impartial assessment. Repeatedly adding features, changing thresholds or trying models against the same validation period can produce overfitting to that period even when no candidate directly fits its labels.
Record the trials and tuning budget, compare with a simple baseline, and preserve a final assessment block. Cross-validation can compare candidates across several development splits, but the splits must match the question. For a past-to-future trading replay, training must use information available before each forecast; randomly shuffling dependent financial observations does not reproduce that learning schedule. Time-aware validation also cannot by itself cure a search that repeatedly exploits the same results.
