Exhaustive Grid Search (GridSearchCV)

GridSearchCV systematically evaluates every combination in a predefined hyperparameter grid using cross-validation, guaranteeing the best result within the grid at the cost of computational time.

How GridSearchCV Works

Given a grid of k hyperparameters with n<sub>i</sub> values each, GridSearchCV trains and evaluates the model for every combination — a total of \\(\\prod n_i\\) fits multiplied by the number of CV folds.

Defining the Parameter Grid

Supply a dictionary where each key is a model parameter name and each value is a list of candidates to try:

<pre><code class="language-python">from sklearn.svm import SVC from sklearn.model_selection import GridSearchCV from sklearn.datasets import load_iris X, y = load_iris(return_X_y=True) param_grid = { "C": [0.1, 1, 10, 100], "kernel": ["rbf", "linear"], "gamma": ["scale", "auto"] } grid_search = GridSearchCV( estimator=SVC(), param_grid=param_grid, cv=5, scoring="accuracy", n_jobs=-1, # use all CPU cores verbose=1 ) grid_search.fit(X, y)</pre>

Inspecting Results

<pre><code class="language-python">import pandas as pd print("Best params: ", grid_search.best_params_) print("Best CV score:", grid_search.best_score_) # Full results as a DataFrame results = pd.DataFrame(grid_search.cv_results_) print(results[["params", "mean_test_score", "rank_test_score"]].sort_values("rank_test_score"))</pre>

Refit and Final Evaluation

By default, GridSearchCV refits the best estimator on the full training set, so grid_search.best_estimator_ is ready to use for prediction immediately.

Nested Cross-Validation for Unbiased Estimates

<pre><code class="language-python">from sklearn.model_selection import cross_val_score # Outer loop gives an unbiased estimate of generalisation performance outer_scores = cross_val_score( GridSearchCV(SVC(), param_grid, cv=3), X, y, cv=5, scoring="accuracy" ) print("Nested CV accuracy: %.3f +/- %.3f" % (outer_scores.mean(), outer_scores.std()))</pre>

Pros and Cons

Grid search is simple and reproducible but scales exponentially — adding one more hyperparameter with just 4 values quadruples the search time.

When to Use Grid Search

Prefer GridSearchCV when your hyperparameter space is small (fewer than ~100 total combinations) or when you need a fully exhaustive, auditable search. For larger spaces, switch to RandomizedSearchCV or Bayesian optimisation.