Skip to main content

LightGBM with scikit-learn

This example uses the bundled diabetes regression dataset and LightGBM's current callback API. Training, early-stopping validation, and final test observations are disjoint.

1. Fit with a validation callback​

import lightgbm as lgb
from sklearn.datasets import load_diabetes
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
X_development, X_test, y_development, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
)
X_train, X_valid, y_train, y_valid = train_test_split(
    X_development,
    y_development,
    test_size=0.20,
    random_state=42,
)

model = lgb.LGBMRegressor(
    objective="regression",
    n_estimators=500,
    learning_rate=0.03,
    num_leaves=15,
    min_child_samples=15,
    subsample=0.9,
    colsample_bytree=0.9,
    reg_lambda=1.0,
    random_state=42,
    n_jobs=1,
    verbosity=-1,
)
model.fit(
    X_train,
    y_train,
    eval_X=X_valid,
    eval_y=y_valid,
    eval_metric="l2",
    callbacks=[
        lgb.early_stopping(stopping_rounds=20, verbose=False),
        lgb.log_evaluation(period=0),
    ],
)

prediction = model.predict(X_test)
print("Best iteration:", model.best_iteration_)
print("Test MAE:", round(mean_absolute_error(y_test, prediction), 2))
print("Test RMSE:", round(root_mean_squared_error(y_test, prediction), 2))
print("Test R-squared:", round(r2_score(y_test, prediction), 3))

assert X_train.shape[0] + X_valid.shape[0] + X_test.shape[0] == X.shape[0]
assert prediction.shape == y_test.shape
assert 0 < model.best_iteration_ <= model.n_estimators

2. Why no scaler appears​

Axis-aligned tree splits do not need standardized inputs. Scaling would not make this LightGBM model more valid and would add another learned artifact. If a combined pipeline contains a scale-sensitive component, fit that transformation only within its training fold.

3. Tuning and early stopping​

The callback watches the validation set and chooses best_iteration_; therefore the validation set is part of model selection. Do not replace it with the test set. Fixed eval_X and eval_y passed through GridSearchCV.fit do not create fold-specific stopping data. During cross-validated tuning, either search a fixed n_estimators or explicitly create stopping data inside each training fold.

4. Evaluation and serving​

MAE, RMSE, and R2R^2 emphasize different error properties; retain a domain-relevant metric and inspect residuals and subgroups. Persist feature order and types, category metadata where applicable, missing-value policy, best_iteration_, and the exact LightGBM build.