LightGBM Theory
LightGBM is a histogram-based gradient-boosted tree library designed for efficient training. Its original paper introduced Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB); not every default model or dataset necessarily exercises those techniques.
1. Histogram-based split finding
Continuous values are bucketed into a finite number of bins. The learner accumulates gradient/Hessian statistics per bin rather than considering every distinct value. This can reduce memory and split-search cost, but max_bin, dataset construction, and categorical representation can affect the fitted model.
2. Leaf-wise growth
LightGBM typically expands the leaf with the largest available gain rather than growing every level uniformly. Leaf-wise growth can reduce training loss quickly, but an unconstrained tree can become deep and data-poor. Control it primarily with num_leaves, min_child_samples/min_data_in_leaf, min_split_gain, and optionally max_depth.
The relation \text{num_leaves}\le2^{\text{max_depth}} is only a capacity comparison with a full depth-wise binary tree, not a rule that makes leaf-wise and depth-wise trees equivalent.
3. GOSS and EFB
GOSS retains observations with large gradients and samples from those with small gradients, with weighting to correct the split-gain estimate. It is associated with the goss data-sampling strategy, not a blanket description of all LightGBM fitting.
EFB bundles sparse features that are rarely nonzero simultaneously, reducing effective dimensionality while controlling bundle conflicts. Its benefit depends on sparsity structure.
4. Categorical features
LightGBM can find category partitions without one-hot expansion. With the Python interface, categorical values are cast to non-negative int32; negative values are treated as missing. pandas unordered categorical columns can be detected automatically. Codes and category semantics must remain stable at inference, and high-cardinality categories still need regularization and leakage review.
5. Objectives and regularization
learning_rate and n_estimators interact with leaf capacity. Row and feature fractions, L1/L2 penalties, minimum leaf observations/Hessian, and split-gain thresholds constrain other dimensions. Specify the objective and evaluation metric rather than assuming an alias has the desired probability or regression semantics.
LightGBM 4.7 uses callbacks such as lightgbm.early_stopping(stopping_rounds=...); the callback monitors metrics on explicit eval_X and eval_y data. The older eval_set argument is deprecated. The test set must not be that stopping set. After stopping, scikit-learn predictions use the best iteration by default through the fitted estimator's iteration metadata.
6. Installation and limits
LightGBM is Apache-2.0 licensed. Wheels are available for common systems, but source and GPU builds require platform-specific compilers and, on macOS source builds, OpenMP setup. Minor versions may contain small breaking changes under the project's effort-versioning policy, so pin the tested version.
Fast training is workload-dependent. Small data, high cardinality, dense versus sparse layout, thread oversubscription, and GPU transfer can change the result. Leaf-wise growth can overfit, and probability outputs may need training-only calibration.
Summary
LightGBM combines histogram split finding with flexible leaf-wise trees and optional sampling/bundling techniques. Its speed and quality are empirical properties of a particular dataset, configuration, and platform—not universal guarantees.