Hyperparameter Tuning for Decision Trees, SVMs, and Other Algorithms

Different model families expose different capacity controls. A useful search space expresses those semantics rather than applying the same rectangular grid to every estimator.

1. Decision trees

Tune max_depth, min_samples_leaf, max_features, and ccp_alpha together. Minimum leaf support and cost-complexity pruning often matter more than chasing a deep maximum. Use the pruning path computed from training data to define candidate ccp_alpha values; do not choose it from the test set.

These parameters constrain different failure modes. Depth limits the longest interaction path, minimum leaf support prevents rules backed by too few training examples, max_features changes which splits are available, and ccp_alpha trades empirical fit against subtree complexity after growth. Similar validation scores can therefore describe trees with very different stability and inference structure.

2. Support vector machines

Scaling belongs inside a pipeline. Search C logarithmically. For RBF SVMs, search gamma jointly with C; for a linear kernel, omit gamma. Kernel SVC scales poorly with sample count, so include LinearSVC or SGDClassifier(loss="hinge") as large-data baselines.

from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)
pipeline = Pipeline([("scale", StandardScaler()), ("model", SVC())])
search = GridSearchCV(
    pipeline,
    param_grid=[
        {"model__kernel": ["linear"], "model__C": [0.1, 1.0, 10.0]},
        {
            "model__kernel": ["rbf"],
            "model__C": [0.1, 1.0, 10.0],
            "model__gamma": [0.01, 0.1, 1.0],
        },
    ],
    scoring="balanced_accuracy",
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    n_jobs=1,
).fit(X_train, y_train)

print("Best parameters:", search.best_params_)
print("Test balanced accuracy:", round(search.score(X_test, y_test), 3))
assert len(search.cv_results_["params"]) == 12

3. K-nearest neighbors

Scale features inside the fold and tune neighbor count, weights, and distance metric. Values of $k$ must be valid for the smallest training fold. Search odd values only if binary vote ties are a real concern; it is not a universal rule. High-dimensional distance concentration can make representation choice more important than fine tuning.

Smaller $k$ permits more local variation and can increase variance; larger $k$ smooths across a wider neighborhood and can erase minority or local structure. Distance weighting changes how sharply influence decays within the selected neighborhood. Treat the metric and representation as part of the model: tuning $k$ cannot repair a feature space in which geometric proximity has no domain meaning.

4. Gradient boosting

Tune learning rate with a sufficient round budget, and tune tree capacity with minimum data/Hessian support. Row/feature sampling and penalties add regularization. The libraries use different controls:

Library	Capacity controls	Current stopping interface
scikit-learn gradient boosting	max_depth/leaf controls on base trees, n_estimators, learning_rate	estimator-specific validation options or staged evaluation
XGBoost 3.4.1	max_depth/max_leaves, min_child_weight, gamma, sampling, penalties	constructor early_stopping_rounds; fit(eval_set=...)
LightGBM 4.7.0	num_leaves, min_child_samples, optional max_depth, sampling, penalties	callbacks=[early_stopping(...)]; fit(eval_X=..., eval_y=...)
CatBoost 1.2.10	depth/grow policy, l2_leaf_reg, sampling, categorical controls	overfitting detector or fit(..., early_stopping_rounds=..., eval_set=...)

Fixed validation arguments passed through a generic scikit-learn search are not automatically fold-local. Search a fixed round count inside CV or write an explicit nested loop with a stopping subset inside each fold.

Across boosters, learning rate and round count define the length and size of additive steps; depth or leaf count controls per-tree interactions; minimum observations or Hessian support prevents weakly supported leaves; and row/feature sampling plus penalties regularize other dimensions. Vendor parameters with different names should be matched by their effect and feasible capacity, not copied numerically. Record the selected iteration in every fold: large variation can reveal an unstable stopping protocol or heterogeneous data.

5. Fair selection across families

Give every family the same outer splits, primary metric, and declared resource budget. Include required preprocessing time and control threads. The winner of inner CV is not the final performance estimate; use outer folds or an isolated test set once the complete procedure is frozen.

For classification, assess ranking, calibration, and a validation-selected operating threshold separately. For regression, choose error measures aligned with deployment costs and inspect residual structure.