Categorical feature encoding converts category identities, such as an exchange, industry or order type, into a representation a predictive model can process. It is part of feature engineering: the representation determines which distinctions the model can learn and which numerical relationships it is encouraged to assume. For a model of execution failure, replacing venue names with integers can accidentally introduce an ordering; replacing them with future failure rates can reveal the answer.
Identity, order or an estimated outcome?
A venue identifier is nominal: B is not twice A, and C need not be closer to B than to A. An ordered rating band has a ranking, but the gaps between bands need not be equal. Encoding must preserve the intended meaning in the downstream model.
| Encoding | What the model receives | Main consequence |
|---|---|---|
| Integer / ordinal | One code per category | A model treating the code as a number may impose order, spacing or threshold groupings. Supply the intended order explicitly when it exists. |
| One-hot | One indicator column per retained category | Separates identities without assigning a scalar rank; many distinct values can create a very wide input. |
| Count / frequency | Category count, or count divided by the fitting sample size | Measures prevalence, without using targets; categories with identical prevalence become indistinguishable on this feature alone. |
| Target | A category-level outcome statistic, often smoothed | Adds supervised information but needs special care over which labels enter each row’s statistic. |
| Learned embedding | A trainable vector selected by category identity | Learns a task-dependent representation; see categorical embedding for the separate lookup mechanism. |
The official OrdinalEncoder and OneHotEncoder references describe the first two representations. With three venues A, B and C, a full one-hot representation gives B the vector . A scalar code supplied to a categorical-aware estimator can instead act only as an identity; the problem arises when an estimator interprets it numerically.
Cardinality means the number of distinct categories. High cardinality makes one-hot width and rare-category estimation relevant, but does not establish that target encoding will predict better. Sparse storage reduces the cost of storing zeros; it does not create observations for rare categories. For an unregularised linear model with an intercept, retaining all category indicators creates exact collinearity. Dropping one reference category is one remedy, not a universal requirement for every learner.
A target mean can carry its own label
Consider four fictional training orders from category A. The binary target is execution failure by a specified assessment horizon: labels are 1, 1, 0 and 0. The full category mean is
Assigning that value to each training row uses the very outcome the row is meant to predict. For the first row, leave-one-out encoding removes its own label and gives
That exclusion is narrower than a leakage guarantee. In this constant-category example, both positive rows receive and both negative rows receive . A rule predicting failure whenever the encoded value is below recovers every training label. The original CatBoost paper, §3.2, explains this leave-one-out failure as well as leakage from full-sample category means. Excluding a label does not make the resulting training representation behave like a new observation.
A separate issue is instability. With a stipulated prior mean from a broader eligible reference set, and smoothing weight , one simple full-category statistic is
Here is the category and its sample count. This mixes the category mean with the prior; it is an illustrative convention, not a software default. It still includes the four rows’ outcomes and is unsuitable as their supposedly unseen training feature. Smoothing controls shrinkage, not information availability. If the prior is estimated, its labels must also respect the fitting boundary.
Original fictional ledger. The bars compare different information sets, not competing forecast performance. Smoothing and leaving out a row do not establish a valid historical information boundary.
Build the encoding inside the research split
Out-of-fold target encoding constructs a training row’s statistic using other permitted folds. In a historical forecasting replay, those folds must respect time: use earlier eligible observations with completed, available labels, not merely other rows. For the first A observation, there may be no known A outcomes at all; the declared prior is then the fallback. Even leave-one-out can include every later outcome. Overlapping outcome horizons can also make an earlier row’s label unavailable at the encoding cutoff, creating look-ahead bias.
The current TargetEncoder documentation distinguishes cross-fitted training encodings from fitting and transforming the same rows. That distinction does not, by itself, certify a chronological replay. Inspect the splitting strategy and label availability. A time splitter can also leave an initial warm-up block without out-of-fold predictions; handle that block explicitly rather than filling it with a statistic fitted on later labels.
At each outer validation boundary, create the training representations using the declared internal procedure, fit the predictor, then transform the held-out block with statistics fitted only on eligible outer-training data. Select encoding methods and smoothing settings within development data. The Time-Ordered Train, Validation and Test Design lesson develops those eligibility and refitting rules; its preview is public and the full lesson requires membership.
Decide what an unseen category means
Freeze the vocabulary, grouping rules and fitted statistics with the model. An unseen category needs a defined response: reject it, use an unknown bucket, map to a supported rare-category group, or apply a prior. A missing field is a separate data condition; combining it with unseen categories is a modelling choice.
With positive smoothing weight and no category observations, the stated target formula returns ; here that is 0.4. In scikit-learn, the documented target-encoder fallback is the fitted global target mean. One-hot unknown handling is configurable: its documented
ignore option yields all zeros. If a reference category is also represented by all zeros after dropping a column, an unknown can become indistinguishable from that reference. Make that behaviour deliberate.Frequency counts, rare-category grouping and vocabulary discovery must also use the permitted fitting information, even though they do not use labels. Record which data fitted each component and monitor new-category incidence: a technically valid fallback may still be weak evidence for a newly opened venue or changed classification scheme.
Sources
- scikit-learn: OrdinalEncoder, class description and category/unknown parameters.
- scikit-learn: OneHotEncoder, indicator representation, collinearity and unknown-category options.
- scikit-learn: TargetEncoder, shrunk means, cross-fitting and unseen-category fallback. Official software references checked 1 November 2026.
- Prokhorenkova et al., CatBoost: unbiased boosting with categorical features, NeurIPS 2018, §3.2, pp. 3–4: target statistics and the leave-one-out counterexample. The order ledger and stipulated smoothing values above are original illustrations.