Introduction to Hyperparameter Tuning

Hyperparameters control a learning procedure but are not fitted directly by its ordinary objective. Examples include a tree's depth, an SVM's C, a regularization strength, and a boosting round count. Choosing them from validation performance is itself learning from data.

Parameters, hyperparameters, and procedures

Model parameters—tree leaf values or regression coefficients—are learned during fit. Hyperparameters configure that fit. The complete procedure also includes preprocessing, feature selection, resampling, early stopping, probability calibration, threshold choice, and sometimes the model family. All choices informed by labels must stay inside the training side of the evaluation boundary.

A valid holdout workflow

1. Reserve a test set according to the deployment unit: IID rows, groups, or future time.
2. Use only development data for exploration and cross-validated search.
3. Select the configuration and any stopping/threshold/calibration rule.
4. Refit the frozen procedure on the allowed development data.
5. Evaluate once on the test set.

When data is too scarce for a single holdout, nested cross-validation estimates the generalization of the search procedure: inner folds select hyperparameters; outer folds evaluate the selected procedure.

Designing a search space

A search space is a scientific assumption, not clerical input. Use valid conditional combinations and scales:

• search regularization strengths and learning rates on logarithmic scales;
• search integer capacity controls over plausible, bounded values;
• avoid meaningless combinations such as RBF gamma for a linear SVM;
• couple boosting learning rate with a sufficient round range;
• constrain leaf capacity with minimum data support;
• include a simple baseline and the documented defaults.

Do not expand the space after inspecting test performance. Record rejected and failed trials so the reported compute budget is honest.

Group controls by the behavior they change:

• capacity: depth, leaf count, basis size, neighborhood size, or network width;
• regularization: penalty strengths, minimum leaf support, dropout, or sampling fractions;
• optimization: learning rate, batch size, solver tolerances, and iteration limits;
• representation: kernel, distance metric, feature transform, or categorical strategy.

This prevents a common category error: increasing an iteration limit may let an optimizer converge, but it does not impose the same inductive bias as changing model capacity. Conditional spaces should also encode which controls exist together. A kernel parameter is irrelevant to a linear SVM, and an early-stopping patience value is irrelevant when stopping is disabled.

Search strategies

Grid search evaluates a Cartesian product and suits a small discrete neighborhood. Random search spends a fixed budget without forcing every dimension onto a coarse grid and is often stronger when only a few dimensions matter. Successive halving allocates more resources to promising candidates under its assumptions. Sequential model-based/Bayesian methods use past trials to propose later ones and can help when evaluations are expensive.

No strategy removes validation noise. The best score among many candidates is optimistically selected, especially on small or unstable datasets.

A useful staged workflow is to begin with documented defaults and a broad random or space-filling search, inspect only training-side diagnostics for invalid bounds or failed regions, and then run a predeclared narrower search if the budget permits. Treat changing the space as another modeling decision: it must not be guided by the final test result. Grid search is most defensible for a genuinely small discrete set; adaptive methods become attractive when individual trials are expensive enough for information from earlier trials to matter.

Choosing a scoring rule

Optimize a predeclared metric aligned with use: ranking, probability quality, thresholded utility, or regression error. Multi-metric search can report several scores, but refit must state which criterion selects the final model. Accuracy is rarely sufficient for imbalanced decisions, and ROC AUC does not measure calibration.

Reproducibility and resources

Fix splitter seeds where randomness is not the subject, record package versions and all candidates, and control nested parallelism. A search with n_jobs=-1 wrapped around estimators that also use all cores can oversubscribe the machine. Compare algorithms using equal configuration counts, wall-clock budgets, or another explicitly justified resource—not the most flattering result for each.

Next steps

The following pages compare grid and random search, show splitters and nested cross-validation, introduce sequential model-based optimization, and give model-specific spaces. The governing rule remains the same: tune on training data; evaluate the frozen procedure outside it.