Bayesian Optimization for Supervised Models

Bayesian optimization and related sequential model-based methods fit a surrogate to observed configuration-performance pairs and use an acquisition rule to choose the next trial. They can spend an expensive budget more selectively than independent grid or random search, but they do not remove validation noise or leakage.

1. Components

• Search space: bounded continuous, integer, categorical, and conditional choices.
• Objective: a validation or cross-validation loss, including preprocessing and failures.
• Surrogate: a model of promising configurations or densities.
• Acquisition/proposal rule: trades exploration against exploitation.
• Budget: initial trials, maximum evaluations, timeout, and parallelism.

Gaussian processes are common for low-dimensional continuous spaces. Hyperopt's Tree-structured Parzen Estimator (TPE) models densities associated with better and worse observations; it is sequential model-based optimization often grouped under the Bayesian-optimization umbrella.

The surrogate is not the supervised model being deployed. It predicts where useful hyperparameters may lie from the limited trial history. The proposal rule must still explore uncertain regions: pure exploitation can repeatedly sample around an early noisy winner, while pure exploration ignores accumulated evidence. Conditional spaces are especially important because the surrogate should not compare inactive parameters as if they had the same meaning in every branch.

2. Hyperopt 0.3 example

The test set is outside Hyperopt's objective. Every trial uses the same stratified training folds, and the NumPy Generator makes the proposal sequence reproducible.

import numpy as np
from hyperopt import STATUS_OK, Trials, fmin, hp, tpe
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import balanced_accuracy_score
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)
folds = StratifiedKFold(n_splits=4, shuffle=True, random_state=42)

space = {
    "max_depth": hp.randint("max_depth", 2, 11),
    "min_samples_leaf": hp.randint("min_samples_leaf", 1, 9),
    "max_features": hp.uniform("max_features", 0.35, 1.0),
}

def objective(parameters):
    model = RandomForestClassifier(
        n_estimators=120,
        max_depth=int(parameters["max_depth"]),
        min_samples_leaf=int(parameters["min_samples_leaf"]),
        max_features=float(parameters["max_features"]),
        random_state=42,
        n_jobs=1,
    )
    score = cross_val_score(
        model,
        X_train,
        y_train,
        scoring="balanced_accuracy",
        cv=folds,
        n_jobs=1,
    ).mean()
    return {"loss": -float(score), "status": STATUS_OK}

trials = Trials()
best = fmin(
    fn=objective,
    space=space,
    algo=tpe.suggest,
    max_evals=12,
    trials=trials,
    rstate=np.random.default_rng(42),
    show_progressbar=False,
)
final_model = RandomForestClassifier(
    n_estimators=120,
    max_depth=int(best["max_depth"]),
    min_samples_leaf=int(best["min_samples_leaf"]),
    max_features=float(best["max_features"]),
    random_state=42,
    n_jobs=1,
).fit(X_train, y_train)
test_score = balanced_accuracy_score(y_test, final_model.predict(X_test))

print("Best parameters:", best)
print("Best CV balanced accuracy:", round(-min(trials.losses()), 3))
print("Test balanced accuracy:", round(test_score, 3))
assert len(trials.trials) == 12

3. Avoid an archived dependency trap

The original page also used scikit-optimize. Its GitHub repository was archived in February 2024 and its hosted stable documentation still reflects an old release line. Existing installations may remain useful, but new production work should not imply active maintenance. Hyperopt 0.3.0 is pinned for this verified example; alternatives should be selected and pinned under their current maintenance and licensing requirements.

4. Best practices

Use enough random or space-filling initial trials, encode conditional choices, and log every outcome. Do not compare an adaptive search with a much smaller random-search budget. Parallel proposals can reduce adaptivity. Noisy cross-validation objectives may need repeated evaluation, but that increases cost.

The best inner score is still selected optimistically. Use nested cross-validation or an untouched test set for the full optimizer. If the search also chooses preprocessing, early stopping, calibration, or thresholds, those steps belong inside the objective and the outer boundary.

5. When sequential optimization is worth the overhead

Use it when one evaluation is expensive, the search space is structured, and enough sequential trials remain after initial exploration for the surrogate to learn. Random search is often a better control for cheap evaluations, very small budgets, highly noisy objectives, or broad spaces whose conditional structure has not yet been designed.

Parallelism changes the trade-off. A large batch of simultaneous proposals cannot use results that are still running, so it gains throughput while losing some adaptivity. Compare adaptive and non-adaptive methods with the same completed-trial or resource budget, preserve failed and timed-out trials, and report optimizer overhead when model fits are cheap.