CatBoost: In-Depth Guide

CatBoost is a gradient-boosted tree library designed to handle categorical features while reducing target leakage in category statistics and prediction shift during boosting.

1. Ordered categorical statistics

Naive target encoding computes a category mean using an observation's own label, leaking target information. CatBoost forms target statistics from prefixes of random permutations: for an observation, only preceding labels contribute. Priors regularize rare categories, and multiple categorical combinations can be constructed.

This mechanism does not prevent all leakage. A target-derived input, duplicated entity across folds, future information, or preprocessing fitted before splitting remains invalid. Declare categorical columns by name or index and preserve their values and schema; arbitrary integer coding without cat_features makes them numeric.

2. Ordered boosting

Standard boosting can reuse each observation's target both to build the current model and to calculate its next gradient, contributing to prediction shift. CatBoost's ordered boosting uses permutation-dependent models so a training example's gradient is based on models that did not train on that example's target in the same way. The production implementation applies approximations for efficiency.

3. Symmetric trees

CatBoost commonly builds symmetric (oblivious) trees: every node at a depth uses the same split condition. This produces regular structure and fast inference, but it is a modeling constraint, not a universal accuracy advantage. depth, min_data_in_leaf availability, and grow_policy interact; parameter support can differ by grow policy and processing unit.

4. Objectives, missing values, and stopping

For binary targets, CatBoostClassifier commonly selects Logloss; multiclass targets select MultiClass under documented conditions. Set loss_function and eval_metric explicitly when reproducibility matters. Probability output and thresholded class output answer different questions.

Numeric missing values are supported with configurable modes; categorical missing values should be represented consistently as category strings rather than relying on float NaN semantics. Validate inference schemas and unseen categories.

With an explicit eval_set, the overfitting detector or early_stopping_rounds can select the iteration. use_best_model=True keeps trees through the best validation iteration. That validation set must come from training data, never the isolated test set.

5. Regularization and computation

Important controls include learning_rate, iterations, depth, l2_leaf_reg, random_strength, row sampling, and categorical-statistic settings. CPU and GPU training support different parameter combinations and are not guaranteed bit-for-bit identical. Tune a bounded, meaningful space and log package version, processing unit, seed, thread count, and feature schema.

CatBoost is Apache-2.0 licensed. Its official Python package supports 64-bit CPython; GPU availability and driver requirements are platform-specific.

Summary

CatBoost's key ideas are ordered categorical statistics and ordered boosting, commonly paired with symmetric trees. They address specific sources of target leakage and prediction shift but do not replace sound splitting, validation, calibration, or schema management.