Assess a model against a predeclared baseline on honest held-out evidence: why 80% accuracy can trail 90%, and why predictive scores do not establish trading profit.
Model evaluation
Model evaluation is the process of measuring how a model performs on a defined task, using an explicit metric, comparison rule and assessment dataset. For a quant, the question is whether a forecast adds useful information under the conditions in which it will be used. A high score alone does not answer that question—and a useful forecast still needs a workable trading policy.
Evaluation happens during development as well as at final assessment. Model selection uses performance estimates to choose a procedure; final assessment asks how that chosen procedure performs on evidence kept outside that choice. Confusing those roles makes a reported result too optimistic.
An 80% score that falls short
Consider ten synthetic, previously unseen cases. The target is a binary label: “positive” could mean a future return above a predeclared cutoff; it is a class name, not a profit. Cases 1–9 are positive and case 10 is negative. All cases have equal weight.
The model predicts positive for cases 1–8 and 10, and negative for case 9. Its confusion matrix is:
Actual label
Predicted positive
Predicted negative
Positive
8 true positives
1 false negative
Negative
1 false positive
0 true negatives
Its accuracy is therefore
Accuracy=casescorrect predictions=108+0=80%.
Suppose a constant-positive rule was fixed before assessment, using development evidence. On these same ten cases it gets nine right: 90% accuracy. The model trails it by 10 percentage points. Both miss the negative case, while the model additionally misses one positive case.
Ten synthetic cases: actual labels are positive for cases 1–9 and negative for 10. The model misses cases 9 and 10, scoring 80%; a constant-positive rule misses only case 10, scoring 90%.
Original synthetic prediction ledger. Blue-filled prediction cells are correct; outlined cells are errors. No market data, trade returns or costs are represented.
This is the effect of class imbalance: the common label supplies a high accuracy score without needing features. Do not choose the baseline's class after inspecting test labels; that would give the comparison information unavailable at the assessment boundary. The example establishes only the two sample scores. Ten observations, with just one negative, do not establish either rule's future predictive skill.
Decide what the score should measure
The metric should follow the task. For hard classifications, report class counts and the confusion matrix alongside accuracy. Precision describes how many positive predictions are correct; recall describes how many actual positives are found. Their importance depends on the consequences of each error. Changing a classification threshold changes those errors, so threshold choice belongs in development.
If the output is a probability, evaluate the probabilities themselves, for example with log loss, and inspect calibration. A hard-label accuracy score discards confidence. For a continuous return forecast, use a loss appropriate to the forecast target and compare with a simple forecast made from the same available information.
No metric removes the need for an appropriate comparator. Keep the prediction horizon, sample, information set and weighting consistent across model and baseline. Report the primary metric chosen beforehand; selecting whichever metric looks best is another research choice.
Preserve the assessment boundary
An out-of-sample row must be outside the fitting and selection decisions being evaluated. That includes preprocessing, feature selection, probability calibration and threshold tuning—not only the final coefficient fit.
For a simulation of historical deployment, fit using information available before each forecast. Labels must have finished and become available before entering training. Where outcome windows overlap assessment outcomes, remove the leaking training observations; a chronological row split alone can leave that overlap intact. Cross-validation must respect the information structure of the task, rather than assume that randomly shuffled financial rows are independent.
Keep a record of the models, features, thresholds and revisions tried. Repeatedly consulting a final test to repair the model makes that test part of development. Overfitting can occur in the selection process even when each candidate was trained without the test labels. Report uncertainty and performance across relevant time blocks; dependence and changing regimes limit what an aggregate score can establish.
From predictions to a trading decision
Prediction errors give no cash-flow ledger. A correct direction on a tiny move may earn less than an incorrect direction loses on a large move. Position sizing, execution delays, turnover, transaction costs, funding and risk constraints determine the economic result.
Evaluate the declared forecast-to-position rule separately, then reconcile it with the predictive evidence. Compare net outcomes under consistent exposure and cost assumptions. The owned Backtest Evaluation lesson covers that report, including benchmarks, drawdowns, concentration and uncertainty; a public preview is available, with membership required for the full lesson.
A defensible conclusion names the task, test design, baseline, metric, search history and limitations. In the ten-case illustration, the defensible conclusion is narrow: the model achieved 80% accuracy and underperformed the predeclared constant-positive comparator on this sample. There is no evidence here of economic profit.
Sources
Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, second edition, §7.2, pp. 219–222: prediction error, model selection and final assessment. Local second-edition text checked; corrected 12th printing, January 2017.
scikit-learn, Metrics and scoring: task-specific losses, baseline comparisons and classification diagnostics.
