Best Practices for Hyperparameter Tuning
1. Define the estimand and split first
State what future observations the score should represent. Choose IID, grouped, or time-aware outer splits before feature engineering. Reserve the test set before exploration and never use it for early stopping, search-space revision, calibration, or threshold selection.
2. Put every learned step inside the fold
Imputation, scaling, encoding, resampling, feature selection, dimensionality reduction, and calibration learn from data. Use a pipeline or an explicit fold loop so each is fitted from fold-training data only. Target encoding needs especially careful out-of-fold or ordered construction.
3. Start with baselines and meaningful defaults
Measure a dummy predictor and a simple regularized model. Include documented defaults in the candidate set unless there is a reason not to. Search high-impact parameters on appropriate scales, honor conditional parameters, and avoid huge spaces that only increase selection noise.
4. Budget fairly
Predeclare candidates, maximum trials, folds, timeout, and early-stopping rules. Compare algorithms under the same configuration count or wall-clock/energy budget and explain the choice. Include preprocessing, failed trials, tuning, and refitting time. Control nested threads to avoid oversubscription.
5. Treat early stopping as tuning
The stopping set chooses the round count. It belongs inside the training process, not in the final test. Reusing one holdout after many experiments can overfit it even if each individual fit is formally separate.
Patience is itself a hyperparameter, and the selected round depends on the metric's noise and evaluation frequency. Save both the best score and best iteration rather than assuming the last fitted iteration is the selected model. For a final refit, document whether you preserve a training-only stopping split, use a robust round count learned from inner folds, or rerun the complete selection procedure on the permitted development data.
6. Use multiple metrics deliberately
Select one primary scorer aligned with use, then diagnose complementary qualities. Ranking metrics do not measure calibration; proper scoring rules do not choose an operating threshold; threshold metrics depend on costs and prevalence. In scikit-learn multi-metric searches, set refit explicitly to the selection criterion.
7. Use nested evaluation when comparing procedures
If the reported number should include hyperparameter selection, use an outer loop. Inner best_score_ is the score that won a competition among candidates and is optimistically selected. Report outer-fold variation and subgroup/temporal behavior rather than only the mean.
8. Log enough to reproduce the result
Record package and hardware versions, data snapshot, split indices or seeds, pipeline, search distributions, every candidate and status, scorer direction, fit/score times, selected parameters, stopping iteration, and final artifact. A random seed aids replay but does not make one split representative.
9. Refit and monitor
After the protocol is validated, refit the frozen procedure on the allowed development data. For boosting, decide how the stopping round is transferred or re-estimated without the test set. At deployment, monitor input schema, drift, missingness, calibration, subgroup errors, latency, and resource use.
The final artifact must include every transformation needed at inference, the feature and category schema, class ordering, threshold or decision rule, and the library versions that load it. A reproducible search log without a reproducible refit is not enough. Conversely, refitting on more data is not automatically beneficial if it silently changes time coverage, group composition, category levels, or the stopping protocol.
Checklist
- Correct outer boundary for deployment.
- Fold-local learned transformations.
- Predeclared primary metric and budget.
- No test-set tuning or stopping.
- Controlled parallelism and complete trial log.
- Final refit rule and monitoring plan.