KNN with scikit-learn

KNN predictions depend directly on distances, so scaling belongs inside the model-selection pipeline. This example tunes $k$ and distance weighting on training folds while keeping the test set untouched.

from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("knn", KNeighborsClassifier()),
])
search = GridSearchCV(
    pipeline,
    param_grid={
        "knn__n_neighbors": [3, 5, 7, 9],
        "knn__weights": ["uniform", "distance"],
        "knn__p": [1, 2],
    },
    scoring="accuracy",
    cv=5,
)
search.fit(X_train, y_train)
y_pred = search.predict(X_test)

print(f"Best settings: {search.best_params_}")
print(f"Test accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(classification_report(y_test, y_pred, digits=3))

p=1 selects Manhattan distance and p=2 Euclidean distance under the default Minkowski metric. weights="distance" gives nearer neighbors more influence. Both choices and $k$ are hyperparameters, so they are selected inside cross-validation rather than against the test result.

For grouped, repeated-measure, or temporal observations, replace the random split and ordinary folds with a group-aware or time-aware strategy. Scaling, feature selection, resampling, and dimensionality reduction must be fitted inside every training fold.

Operational trade-offs

Fitting mainly stores the training data and builds any search structure. Prediction must find neighbors and can therefore cost substantial time and memory as the reference set grows. Tree-based neighbor search can help in suitable low-dimensional spaces; brute-force search is often competitive in high dimensions. Validate latency and memory on representative data rather than relying on a universal complexity claim.