Skip to main content

Logistic Regression Common Mistakes and Best Practices

1. Assuming probabilities are linear in the features​

Logistic regression makes the log-odds linear in the supplied features. The probability curve is sigmoid-shaped, and the default binary decision boundary is the hyperplane where the fitted probability equals 0.5. Interactions, splines, or other basis functions can create nonlinear boundaries in the original feature space while remaining linear in transformed features.

2. Scaling or resampling before validation​

Fit scalers, imputers, feature selectors, encoders, and resampling procedures inside each training fold. A pipeline prevents test information from influencing learned means, variances, vocabularies, or selected features.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
pipeline = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1_000),
)
search = GridSearchCV(
    pipeline,
    {"logisticregression__C": [0.01, 0.1, 1.0, 10.0]},
    scoring="neg_log_loss",
    cv=5,
)
search.fit(X_train, y_train)
print(search.best_params_)
print(f"Negative test log loss: {search.score(X_test, y_test):.3f}")

Here GridSearchCV.score uses the search object's configured negative log-loss scorer, not accuracy. This distinction is important: estimator .score() and a search's .score() can have different semantics.

3. Treating 0.5 as a universal threshold​

The model estimates a score or probability, while the threshold encodes an action. Choose the threshold on validation data from error costs, capacity, or policy constraints. Report class-specific precision/recall and the confusion matrix at that threshold. Do not repeatedly tune against the test set.

4. Reporting only accuracy or ROC AUC​

Accuracy depends on class prevalence and threshold. ROC AUC measures ranking across thresholds and can look reassuring when positive cases are rare. Average precision or a precision-recall curve is often more informative for rare positive classes. When probability values matter, assess log loss, Brier score, and calibration as well as discrimination.

5. Misreading regularization and coefficients​

scikit-learn regularizes logistic regression by default. C is the inverse of regularization strength: smaller values regularize more. Penalty/solver combinations are constrained by the current API. Coefficients act on log-odds, depend on feature units, and represent conditional associations—not causal effects. Correlated features can make individual coefficients unstable.

6. Ignoring separation and distribution shift​

Complete or near separation can drive unregularized coefficient estimates toward extreme values. Regularization helps numerically but does not make unsupported inference valid. After deployment, shifts in prevalence or feature relationships can damage thresholds and calibration even when ranking changes little.