CatBoost is an open-source library for gradient-boosted decision trees, with built-in processing of categorical features. A credit-risk model can therefore combine numerical ratios with categories such as industry or region without treating arbitrary category codes as a numerical ranking. Its handling of labels during training is especially relevant when a seemingly useful category feature could reveal the outcome it is meant to predict.
Two mechanisms that share an ordering principle
The original CatBoost paper, §§3–5, distinguishes:
- Ordered category statistics: construct a training row's target-based category feature from earlier rows in a training permutation. Its own label is excluded. This addresses target leakage through the feature.
- Ordered boosting: calculate a row's training gradient using a supporting prediction based on an earlier prefix of the permutation. This addresses prediction shift caused by evaluating training gradients against predictions already influenced by the same targets. CatBoost uses an efficient implementation of this principle.
These are separate mechanisms. The paper's Plain boosting mode still has ordered category statistics; the word Plain does not mean that categories must be encoded externally.
An encoding that cannot read its own answer
Consider a deliberately small illustration of an ordered target statistic. Each observation has a sector category and a binary target: default within a hypothetical one-year horizon is , otherwise . Fix the prior at , independently of these observations, and give it smoothing weight .
For a row with category , define the toy feature
where is the number of earlier rows with category in the chosen permutation, and is the sum of their labels. “Earlier” here means earlier in the permutation, rather than earlier on the calendar.
Take two training rows, both category A, in the stipulated order below. Encode each row before adding its label to the history.
| Permutation position | Own label | Earlier A labels | Feature supplied to this row |
|---|---|---|---|
| First | None | ||
| Second |
The second feature is even though its own label is . Changing that label would leave its feature unchanged. The smoothing has a simple independent interpretation: the fixed prior contributes one positive and one negative pseudocount. The first row sees just those pseudocounts; the second sees them plus the earlier positive label.
Original exact toy ledger for a binary default target over a hypothetical one-year horizon. The prior and smoothing weight are stipulated. This illustrates one ordered-statistic idea, rather than a fitted CatBoost model, its default settings or its full categorical-feature pipeline.
For a held-out observation, target-dependent category statistics use training labels, rather than held-out answers. The CatBoost FAQ explains why those features can use a longer history at validation or prediction time than an individual training row had available.
What to check in an actual fit
Built-in categorical processing includes more than the toy mean above. The categorical-feature documentation describes category combinations and one-hot treatment for small category sets in supported modes. The transformation documentation lists several statistic types, including a counter that does not depend on labels. Do not assume every category receives the same target mean.
The current parameter reference offers
boosting_type="Ordered" and "Plain". Defaults depend on the processing unit, dataset size and learning mode; ordered boosting is not a universal default. Record the library version and actual settings used in the experiment.For financial forecasting, an artificial training permutation does not establish a valid test of future performance. Features must reflect information available at the forecast timestamp. Training labels must have finished their outcome horizons before they are used; overlapping label intervals can also require purging. Keep tuning separate from the final test period to limit overfitting and look-ahead bias.
The documented
has_time setting preserves input order instead of applying random permutations in categorical transformation and tree-structure selection. It does not check when a label became observable or design a train/test split. Sorting by observation start time can still put an unfinished earlier label into the information set.For this separate validation problem, Finance-aware validation develops a label-interval exclusion ledger and distinguishes purging from an embargo. The full lesson requires membership and has a public preview.
Sources
- Prokhorenkova, Gusev, Vorobev, Dorogush and Gulin, CatBoost: unbiased boosting with categorical features, NeurIPS 2018, §§3.2, 4.2 and 5, PDF pp. 3–6.
- CatBoost, Categorical features and Transforming categorical features to numerical features.
- CatBoost, FAQ, on training versus validation/prediction category statistics.
- CatBoost, Common training parameters,
boosting_typeandhas_time; checked 27 October 2026. - Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, second edition, §§10.10.2–10.10.3, pp. 359–361, for the gradient-tree-boosting background.