Skip to main content

Support Vector Machines with scikit-learn

This example keeps all three Iris classes to show scikit-learn's real multiclass behavior. SVC internally fits one-versus-one binary classifiers; its default decision_function_shape="ovr" only presents the resulting scores in a convenient (samples, classes) shape.

SVMs are not scale invariant. Putting StandardScaler and SVC in one Pipeline ensures that every cross-validation fold learns means and variances from that fold's training partition only. Scaling the full dataset before GridSearchCV would leak validation-fold information.

2. Tune the coupled RBF parameters​

For an RBF kernel, C trades a smoother, more regularized boundary against training violations, while gamma controls how local each training point's influence is. Search both on logarithmic ranges and compare against a linear kernel rather than assuming nonlinear is better.

from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data,
    iris.target,
    test_size=0.25,
    random_state=42,
    stratify=iris.target,
)

pipeline = Pipeline(
    [
        ("scale", StandardScaler()),
        ("svc", SVC(decision_function_shape="ovr")),
    ]
)
search = GridSearchCV(
    pipeline,
    param_grid=[
        {"svc__kernel": ["linear"], "svc__C": [0.1, 1.0, 10.0]},
        {
            "svc__kernel": ["rbf"],
            "svc__C": [0.1, 1.0, 10.0],
            "svc__gamma": [0.01, 0.1, 1.0, "scale"],
        },
    ],
    scoring="balanced_accuracy",
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    n_jobs=1,
)
search.fit(X_train, y_train)

svm = search.best_estimator_
y_pred = svm.predict(X_test)
scores = svm.decision_function(X_test)
print("Best parameters:", search.best_params_)
print("Test accuracy:", round(accuracy_score(y_test, y_pred), 3))
print("Decision-score shape:", scores.shape)
print("Support vectors per class:", svm.named_steps["svc"].n_support_)
print("Confusion matrix:\n", confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, target_names=iris.target_names))

assert X_train.shape == (112, 4)
assert X_test.shape == (38, 4)
assert scores.shape == (38, 3)
assert y_pred.shape == y_test.shape

3. Scores are not probabilities​

decision_function returns signed margins or transformed multiclass scores, not calibrated probabilities. In scikit-learn 1.9, SVC(probability=True) is deprecated; when probabilities are required, calibrate an SVC with CalibratedClassifierCV using training data only. Calibration adds another validation layer and can still be unreliable with very small samples.

Select any operational threshold from validation data according to the cost of false positives and false negatives. Do not choose it after inspecting the test set.

4. Scale and deployment limits​

Kernel SVC fit time grows at least quadratically with sample count and may be impractical beyond tens of thousands of observations. For large, approximately linear problems, compare LinearSVC or SGDClassifier(loss="hinge"). For nonlinear structure at scale, evaluate explicit kernel approximation before a linear estimator.

Keep the fitted scaler, classifier, feature order, and class mapping together. At inference, the same training-fitted transformation must be applied to every new observation.