Grid Search vs. Random Search for Hyperparameter Tuning

Grid search evaluates every combination in a finite grid. Random search samples a fixed number of combinations from distributions. Comparing them fairly requires the same data, splitter, scorer, estimator pipeline, and candidate budget.

1. Geometry of the search

A grid with $k$ values in each of $d$ independent dimensions evaluates $k^d$ candidates. It repeatedly tests every value of an unimportant dimension. Random search can explore more distinct values of influential continuous parameters for the same budget, but it can miss narrow regions and does not use outcomes from earlier trials.

Use logarithmic distributions for quantities whose meaningful changes are multiplicative. Use conditional spaces rather than evaluating irrelevant parameters. A small grid is useful for exact reproducibility or a final local comparison after the space was defined without test feedback.

2. Equal-budget example

Both searches below evaluate nine candidates on the same five folds. Scaling remains inside the pipeline, so each fold learns it from fold-training rows only.

from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)
pipeline = Pipeline(
    [("scale", StandardScaler()), ("svc", SVC(kernel="rbf"))]
)
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

grid = GridSearchCV(
    pipeline,
    param_grid={
        "svc__C": [0.1, 1.0, 10.0],
        "svc__gamma": [0.001, 0.01, 0.1],
    },
    scoring="roc_auc",
    cv=folds,
    n_jobs=1,
)
random = RandomizedSearchCV(
    pipeline,
    param_distributions={
        "svc__C": loguniform(0.05, 20.0),
        "svc__gamma": loguniform(0.0005, 0.2),
    },
    n_iter=9,
    scoring="roc_auc",
    cv=folds,
    random_state=42,
    n_jobs=1,
)
grid.fit(X_train, y_train)
random.fit(X_train, y_train)

print("Grid CV AUC:", round(grid.best_score_, 3), grid.best_params_)
print("Random CV AUC:", round(random.best_score_, 3), random.best_params_)
print("Grid test AUC:", round(grid.score(X_test, y_test), 3))
print("Random test AUC:", round(random.score(X_test, y_test), 3))

assert len(grid.cv_results_["params"]) == 9
assert len(random.cv_results_["params"]) == 9

3. Interpret the comparison

The winning test score is descriptive here; choosing the strategy after inspecting those two test scores would tune on the test set. In a real comparison, choose the search protocol in advance or place the protocol itself inside an outer evaluation.

Cross-validation standard deviations describe variation across overlapping folds, not a confidence interval for deployment performance. Repeated or nested evaluation and domain-aware splits provide stronger evidence.

An equal candidate count is one fair budget, but not the only one. Candidate costs can vary sharply with depth, sample count, kernel choice, convergence, or early termination. A wall-clock budget answers a different operational question, and a fixed resource budget such as total fitted trees can answer another. State which quantity is held equal, include failed candidates, and avoid presenting a comparison as fair under every cost definition.

Random search gains coverage only when its distributions encode plausible scales. Uniform sampling of a regularization value across many orders of magnitude concentrates trials at the largest magnitudes; a log-uniform distribution represents multiplicative uncertainty more directly. Conversely, categorical choices and a few policy-approved values are often clearer as an explicit list.

4. Practical choice

Use a grid for a small, genuinely discrete space. Use random search as a robust first pass for wider continuous or mixed spaces. Escalate to adaptive methods only when each evaluation is expensive enough to justify their complexity, and always count failed and early-terminated trials in the budget.

Do not run a large grid merely because it is exhaustive within its listed values: it says nothing about values between or outside them. Likewise, one seeded random search is not proof that random search is intrinsically superior. The comparison concerns two finite experimental designs, so preserve their sampled candidates and results in the experiment record.