Skip to content

Hyperparameter

A hyperparameter configures a learning procedure; fitted parameters are learned within it. See the distinction through tree depth, chronological validation and scheduled refitting.

A hyperparameter is a setting that configures a learning procedure rather than being estimated as fitted model state within an individual fit. For a return forecast, changing that setting can change how much history the model remembers, how flexible it is or how strongly it resists fitting noise. It therefore belongs in the research record alongside the fitted model.
The boundary is the individual fit. A hyperparameter can be selected using data through an outer hyperparameter-tuning procedure; it need not be a number chosen by hand or fixed forever. The scikit-learn tuning guide uses this distinction between estimator settings and values learned within an estimator.

A depth limit is not a split threshold

Consider a hypothetical decision tree that forecasts a future return from features available at the forecast time. A researcher specifies candidate settings max_depth=2 and max_depth=5 before their respective fits. For the illustration, use max_depth=2. This choice demonstrates a configuration; no validation results establish that it is the better choice.

Depth is measured along the longest path from the root to a leaf, counting edges. The cap limits that path; it does not require the fitted tree to reach the cap. During fitting, the training data determine which features and thresholds split the observations and what prediction each leaf supplies. Those learned choices form fitted model state. In statistical terminology they are model parameters, even though a software API may also call its configurable inputs “parameters.”

Chosen for the learning procedureLearned within a particular fit
A tree's maximum permitted depthIts split features, split thresholds and leaf predictions
A regression's regularisation strengthIts fitted coefficients under that penalty
A neural network's learning-rate scheduleIts fitted weights after optimisation

The distinction follows the procedure, not the appearance of the number. A threshold estimated by a tree's training algorithm is fitted state. A threshold manually varied to improve a strategy's validation score is a research setting being selected outside that fit. A regularisation strength can similarly be selected in an outer search while coefficients are estimated in the inner fit.

Hypothetical candidates max_depth 2 and 5 are specified before fitting. Training learns split features, thresholds and leaf predictions. Validation selects settings before a frozen procedure faces a later final test.
Hypothetical candidates max_depth 2 and 5 are specified before fitting. Training learns split features, thresholds and leaf predictions. Validation selects settings before a frozen procedure faces a later final test.
Original hypothetical procedure diagram. Depth 2 is an illustration, not a reported winner. No market data or fitted performance scores are used.

Selection consumes evidence

Training estimates fitted state for a candidate configuration. Validation compares candidates. A final test assesses the chosen procedure on outcomes withheld from that selection. A validation score used to choose depth is therefore not an untouched final assessment of that choice. Repeated searches can overfit the validation evidence too.
The comparison criterion should match the intended task. Forecast error measures predictive accuracy; a trading objective also depends on position rules, transaction costs and risk. Choosing a different metric after seeing which candidate it favours adds another selection decision. More depth offers more flexibility, not a guarantee of better future forecasts or net trading performance.

A frozen procedure can still refit

For financial forecasts, reproduce the information available at each historical decision. Use earlier eligible observations to fit and later development observations to validate. A dated training row is insufficient if its future-return label had not finished when the fit occurred. Preprocessing that learns from data must also obey the declared fitting boundary. Ordinary shuffled folds do not reproduce this historical learning sequence; chronological ordering still needs checks for overlapping outcomes and look-ahead bias.

Suppose the intended procedure refits monthly with the selected depth cap. New observations can then change thresholds and leaf predictions while the hyperparameter remains unchanged. Freezing the procedure before final testing means fixing its permitted information, fitting schedule and selection rules. It does not require preserving the same fitted tree throughout the test.

A system may also retune on a declared schedule. If that is the intended live behaviour, specify and replay the whole tuning rule using only information available at each update. Changing the rule after inspecting final-test outcomes makes those outcomes part of development evidence.

Time-Ordered Train, Validation and Test Design works through label eligibility and walk-forward refitting. Its public preview describes the lesson; the full lesson requires membership.

Sources

  • scikit-learn, Tuning the hyper-parameters of an estimator, §3.2: estimator settings and outer searches.
  • scikit-learn, DecisionTreeRegressor: max_depth, learned tree state and the depth definition. The candidate settings here are hypothetical.
  • scikit-learn, Cross-validation: evaluating estimator performance, introduction and §3.1.2.6: validation versus final testing and time-series splitting.
  • Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2nd edition, §7.2, p.222; local corrected 12th printing, January 2017: tuning parameters and model selection versus model assessment. The chronological financial procedure here additionally follows the owned lesson linked above.
Last updated
  • machine learning
  • model selection
  • validation
  • research process
Keep connecting the ideas
← Back to the glossaryBack to top ↑