A test set assesses a frozen model or research procedure on held-out data. See how chronological splits, preprocessing and repeated test reuse affect the evidence.
Test set
A test set is data held aside to assess a model or research procedure after its choices have been fixed. For a quant strategy, it asks how the selected procedure behaved on observations that did not help choose it. A favourable result is more informative when the test never influenced the feature set, model settings or trading rules.
Consider 100 synthetic observations ordered by decision time. Assign observations 1–60 to training, 61–80 to validation and 81–100 to the final test. That reserves 20 test observations while the first 80 support development. These are illustrative allocations, not a recommended universal ratio; dataset size, forecast horizon and the evaluation question determine a useful design.
Synthetic 100-row chronological assignment: training observations 1–60, validation 61–80, final test 81–100. Freeze research choices before opening the test. Changes based on the test turn it into development evidence.
Original synthetic assignment, with block widths proportional to row counts. The split is illustrative; actual training and assessment eligibility also depends on feature and label availability.
Fit the candidates using eligible training observations. Choose the model and settings using validation evidence. Before opening the test, record the target, features, preprocessing, refitting schedule, trading costs, primary metric and decision rule. Then assess the frozen procedure on the test block, using only information available at each simulated decision.
A final refit on eligible training and validation data can be part of that frozen procedure. So can scheduled refitting as new outcomes arrive. The update rule must be specified in advance and replayed honestly; a final test need not mean one coefficient vector held constant forever.
Keep the whole pipeline behind the boundary
A model can avoid fitting to test labels and still suffer look-ahead bias. For example, scaling features using a mean estimated from the full history lets the test-period distribution influence the inputs. Fit data-dependent preprocessing on the permitted training window, then apply those fitted transformations to held-out observations. Imputation and feature selection need the same discipline. The scikit-learn leakage guide explains this distinction.
Chronological row numbers are only an assignment. A training row near a boundary may have a future-return label that finishes inside the held-out period. That outcome was unavailable at the training cutoff. Exclude unfinished labels and remove training-label overlap with the evaluation block where required. In strictly past-only training there are no post-test training observations to embargo; a pre-test gap may still be justified by availability or dependence. A random shuffle does not reproduce a strategy trained on the past to predict the future.
For the full timing rules, Time-Ordered Train, Validation and Test Design develops eligibility checks and walk-forward schedules. The full lesson requires membership; its opening and outline have a public preview.
What happens after you look?
If the final test disappoints and you adjust a signal threshold, drop a feature or select a better-looking strategy using that result, the period now informs development. Re-running it does not produce a fresh final test. Record the change and seek fresh assessment data under a declared design. This is overfitting through the research process, even when each model fit excludes the test rows.
A clean test is still a finite historical sample. Its 20 observations in this example need not be 20 independent bets, and they may cover only a narrow market regime. Report the observation count, dependence, costs and uncertainty alongside the chosen metric. A good test result supports a claim about the evaluated procedure under those conditions; it does not establish permanent profitability.
