What is overfitting in machine learning?

May 18, 2026··
11 min read
Open full tree
What is overfitting in machinelearning?Fundamentals of Machine LearningUnderstanding OverfittingPreventing OverfittingPractical ApplicationsAdvanced Concepts

What is Overfitting in Machine Learning?

Abstract
Overfitting is one of the central problems in machine learning: a model fits the training data so closely that it captures noise and idiosyncrasies of the training set rather than the true underlying patterns, leading to poor performance on new (unseen) data. This article gives a deep, practical and theoretical treatment of overfitting: definitions, historical background, mathematical foundations, common causes, diagnostics, prevention strategies, practical code examples, modern phenomena (e.g., double descent and implicit regularization in deep nets), and a checklist for practitioners.

Table of contents

  • Overview and intuitive definition
  • Historical context
  • Formal/theoretical foundations
    • Empirical risk vs expected risk
    • Generalization error and finite-hypothesis bounds
    • VC dimension and structural risk minimization
    • Rademacher complexity and stability
    • Bias–variance decomposition
    • Double descent and modern deep learning
  • Causes of overfitting
  • Practical examples and reproducible demos (code)
    • Polynomial regression demo (scikit-learn)
    • Neural network demo (Keras) using early stopping and dropout
  • How to detect overfitting
    • Learning curves and train/validation gap
    • Cross-validation patterns
    • Residuals, calibration, and confidence
  • How to prevent or mitigate overfitting
    • Data-level strategies
    • Model-level strategies
    • Algorithm-level strategies
    • Evaluation and model selection techniques
  • Practical checklist: diagnosing and fixing
  • Current state of research and open problems
  • Future implications
  • Selected references

Overview and intuitive definition

  • Intuitive definition: Overfitting occurs when a model has learned patterns that are specific to the training dataset (including noise or spurious correlations) rather than patterns that generalize to other data sampled from the same source.
  • Manifestation: Very low training error paired with substantially higher validation/test error. The model gives high confidence predictions for training cases but fails on new examples.

Historical context

  • The term and phenomenon date back to classical statistics and curve fitting (e.g., polynomial interpolation). In statistical learning theory, work by Vladimir Vapnik and others formalized the notion of generalization and introduced constructs such as VC dimension and structural risk minimization to reason about overfitting.
  • Practical machine learning communities historically used cross-validation, holdout sets, and regularization early on to combat overfitting. The rise of deep learning reawakened theoretical interest because massively overparameterized neural networks often fit training sets perfectly yet can still generalize — giving rise to new theory and phenomena (e.g., interpolation regimes and double descent).

Theoretical foundations

  1. Empirical risk vs expected risk
  • Let X × Y be the input/output space, and P(x, y) a data distribution. For a hypothesis h ∈ H, define the loss ℓ(y, h(x)) (e.g., squared error, 0–1 loss).
  • Expected risk (true/generalization error): R(h) = E_{(x,y)∼P}[ℓ(y, h(x))]
  • Empirical risk (training error on sample S of size n): R_emp(h) = (1/n) ∑_{i=1}^n ℓ(y_i, h(x_i))
  • Overfitting arises when R_emp(h) is low but R(h) is substantially larger.

  1. Finite-hypothesis bound (Hoeffding-type) For a finite hypothesis class H, with probability at least 1 − δ (over random draw of S), R(h) ≤ R_emp(h) + sqrt((ln|H| + ln(2/δ)) / (2n)) This shows: for fixed sample size n, larger hypothesis classes (larger |H|) increase the bound on generalization error.

  2. VC dimension and structural risk minimization

  • VC dimension (d_vc) is a capacity measure for binary classifiers. With probability at least 1 − δ, R(h) ≤ R_emp(h) + O( sqrt( (d_vc log(n/d_vc) + log(1/δ)) / n ) ).
  • Structural risk minimization (SRM) advocates ordering hypothesis classes by complexity and selecting a model that balances R_emp and complexity to control generalization error.
  1. Rademacher complexity and algorithmic stability
  • Rademacher complexity gives a data-dependent capacity measure. Lower Rademacher complexity implies tighter generalization bounds.
  • Algorithmic stability measures how sensitive an algorithm’s output is to small changes in the training set; more stable algorithms generalize better.
  1. Bias–variance decomposition (for regression with squared loss)
  • Prediction error decomposes (for squared error) into: E[(y − ŷ)^2] = (Bias[ŷ])^2 + Variance[ŷ] + Irreducible noise
  • Overfitting typically comes from high variance: model fits noise, predictions vary a lot across training samples.
  1. Double descent (modern phenomenon)
  • Classical bias–variance suggests test error monotically decreases then increases with model complexity. However, modern overparameterized models (e.g., deep nets) can show "double descent": test error decreases, then increases around the interpolation threshold (when the model can fit training data perfectly), and then decreases again as capacity increases further.
  • This motivated new theoretical work on implicit regularization and the role of optimization (e.g., SGD) in selecting solutions that generalize.

Causes of overfitting

  • Excess model capacity relative to the amount of data (too many parameters or very flexible model).
  • Insufficient or non-representative training data (small n, sampling bias).
  • Noisy labels and outliers in training set.
  • Data leakage: information from validation/test leaks into training (e.g., using target-derived features).
  • Overly long training (in iterative learners) without proper validation/early stopping.
  • Overly complex feature set, spurious features, or highly correlated features.
  • Using inappropriate cross-validation (e.g., random CV for time-series).

Examples & reproducible demos

A. Polynomial regression example (demonstrates classic overfitting)

Python (scikit-learn & matplotlib) — ready-to-run example:

Python
1import numpy as np 2import matplotlib.pyplot as plt 3from sklearn.preprocessing import PolynomialFeatures 4from sklearn.linear_model import LinearRegression 5from sklearn.metrics import mean_squared_error 6from sklearn.model_selection import train_test_split 7 8# Generate synthetic data 9rng = np.random.RandomState(0) 10n_samples = 30 11X = np.linspace(-3, 3, n_samples)[:, None] 12y_true = np.sin(X).ravel() 13y = y_true + rng.normal(scale=0.3, size=n_samples) # noisy targets 14 15X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1) 16 17degrees = [1, 3, 9] 18train_errors = [] 19test_errors = [] 20 21plt.figure(figsize=(12, 4)) 22for i, deg in enumerate(degrees): 23 poly = PolynomialFeatures(degree=deg) 24 X_train_p = poly.fit_transform(X_train) 25 X_test_p = poly.transform(X_test) 26 model = LinearRegression().fit(X_train_p, y_train) 27 y_train_pred = model.predict(X_train_p) 28 y_test_pred = model.predict(X_test_p) 29 train_errors.append(mean_squared_error(y_train, y_train_pred)) 30 test_errors.append(mean_squared_error(y_test, y_test_pred)) 31 32 # Plot fit 33 X_plot = np.linspace(-3, 3, 200)[:, None] 34 y_plot = model.predict(poly.transform(X_plot)) 35 36 plt.subplot(1, 3, i+1) 37 plt.scatter(X_train, y_train, label='train') 38 plt.scatter(X_test, y_test, label='test') 39 plt.plot(X_plot, y_plot, color='red', label=f'deg {deg}') 40 plt.title(f'degree {deg}\ntrain MSE={train_errors[i]:.3f}, test MSE={test_errors[i]:.3f}') 41 plt.legend() 42 43plt.show()

Interpretation: low-degree polynomial (underfitting) → high train/test error; high-degree polynomial (overfitting) → very low train error, high test error.

B. Neural network demo with early stopping and dropout (Keras)

Python
1import tensorflow as tf 2from tensorflow.keras import layers, models, regularizers 3from tensorflow.keras.callbacks import EarlyStopping 4 5# Assume X_train, y_train, X_val, y_val prepared (e.g., MNIST small subset) 6model = models.Sequential([ 7 layers.Dense(512, activation='relu', kernel_regularizer=regularizers.l2(1e-4)), 8 layers.Dropout(0.5), 9 layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(1e-4)), 10 layers.Dropout(0.5), 11 layers.Dense(num_classes, activation='softmax') 12]) 13 14model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) 15es = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True) 16 17history = model.fit(X_train, y_train, validation_data=(X_val, y_val), 18 epochs=100, batch_size=128, callbacks=[es])

Interpretation: Early stopping prevented the model from continuing to fit noise; dropout and L2 reduce capacity and improve generalization.

How to detect overfitting

  1. Train vs validation/test performance gap
  • The simplest and most robust signal: training accuracy low/close to zero error but validation/test error much larger.
  1. Learning curves
  • Plot error as a function of training set size or number of epochs:
    • Overfitting: training error low, validation error high; validation error decreases with more data.
    • Underfitting: both errors high; adding model capacity helps.
  1. Cross-validation results
  • Large variance across folds suggests high variance models (prone to overfitting).
  1. Residual analysis (regression)
  • Structured residual patterns and large residuals on new data imply overfitting to training noise.
  1. Prediction confidence and calibration
  • Overfitting models (esp. overconfident classifiers) may show poor calibration: predicted probabilities don't match empirical frequencies.
  1. Sensitivity to perturbation
  • Overfit models change much if you remove a few training examples — indicates high variance.

How to prevent or mitigate overfitting

Broad categories: get more/better data; reduce model capacity or enforce constraints; use algorithmic regularization; carefully evaluate using proper CV.

Data-level strategies

  • Get more labeled data (most direct remedy).
  • Data augmentation (images: flips, crops, color jitter; text: paraphrase, back-translation).
  • Label cleaning and outlier detection.
  • Synthetic data generation (careful to avoid distribution shift).

Model-level strategies

  • Use simpler models or reduce model capacity:
    • For trees: limit depth, min_samples_leaf, pruning.
    • For linear models: restrict number of features or use feature selection.
    • For neural nets: smaller architectures.
  • Regularization:
    • L2 (weight decay) adds λ||w||_2^2 to objective.
    • L1 encourages sparsity (feature selection).
    • Elastic net = α L1 + (1−α) L2.
  • Dropout for neural nets: randomly zero units during training, acts like model averaging.
  • Batch normalization can improve generalization (and speed training).
  • Label smoothing (classification): prevents overconfident outputs.

Algorithm-level strategies

  • Early stopping: stop training when validation error stops decreasing.
  • Ensembling:
    • Bagging reduces variance (e.g., Random Forests).
    • Model averaging across different initializations, architectures.
    • Stacking combines different base learners.
  • Bayesian approaches:
    • Place priors over parameters (regularization is equivalent to MAP under priors).
    • Full Bayesian posterior inference can capture uncertainty and prevent overconfident overfitting.
  • Semi-supervised learning / transfer learning:
    • Use pretrained models and fine-tune; transfer can reduce overfitting on small datasets.

Evaluation and model selection techniques

  • Use a properly held-out test set reserved until final evaluation.
  • Use k-fold cross-validation (stratified for classification).
  • For time series, use forward-chaining (rolling) CV to respect temporal ordering.
  • Nested cross-validation for tuning hyperparameters to avoid selection bias.
  • Be careful of data leakage: avoid feature engineering that uses any information from the test fold.

Pruning and compression

  • Decision tree pruning (post- or pre-pruning).
  • Model compression/pruning for neural nets can remove unnecessary weights.
  • Knowledge distillation: train a smaller student model on soft teacher outputs.

Adversarial and robustness considerations

  • Training on adversarial examples or mixup (interpolations between training samples) can improve robustness and sometimes reduce overfitting.

Practical diagnostics: what to look at (checklist)

  1. Plot train and validation loss/accuracy over epochs — do you see a gap?
  2. Plot train vs test error for different model complexities (e.g., polynomial degree or network size).
  3. Compute learning curves: error vs training set size.
  4. Use k-fold CV to estimate variance across folds.
  5. Check calibration for classifiers (reliability diagrams).
  6. Test sensitivity: remove a small fraction of training data; does model change drastically?
  7. Look for data leakage: are feature creation steps using target information?
  8. Examine feature importances — are spurious or irrelevant features driving predictions?

Fixes once diagnosed

  • If variance-high: reduce capacity, add regularization, get more data, use ensembling.
  • If both train and val errors high: increase capacity, choose richer model or more informative features.
  • If validation error reduces with more data: collect more data if feasible.

Current state of research and open problems

  • Understanding generalization in overparameterized models: Why do deep nets generalize despite having many more parameters than training samples? Research focuses on implicit biases of optimization algorithms (SGD), flat vs sharp minima, and properties of interpolating solutions.
  • Double descent and interpolation: characterizing regimes and prediction of when double descent occurs.
  • Distribution shift and out-of-distribution (OOD) generalization: overfitting to the training distribution can make models brittle under shift.
  • Robust generalization under adversarial attacks: adversarial training can reduce accuracy/generalization tradeoffs but remains an active area.
  • The role of data quality, label noise, and how to build models robust to noisy labels.
  • Provable guarantees for modern architectures: connecting capacity measures (VC, Rademacher) to deep networks remains challenging.

Future implications

  • As models become larger and datasets more complex, overfitting remains central to deploying reliable ML systems. Tools and theory that better predict generalization and control it under distribution shift will be increasingly important.
  • In safety-critical and high-stakes domains, minimizing overfitting is not only about predictive accuracy but about calibration, reliability, and interpretable uncertainty estimates.
  • Techniques like unsupervised pretraining and transfer learning reduce labeled-data needs and can mitigate overfitting, but raise questions about transfer bias and domain mismatch.

Selected references and further reading

  • Vapnik, V. (1998). Statistical Learning Theory.
  • Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning.
  • Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences.
  • Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., & Srebro, N. (2018). Towards understanding the role of over-parametrization in generalization of neural networks.
  • Keskar, N. S., et al. (2017). On large-batch training for deep learning: Generalization gap and sharp minima.

Concluding summary

Overfitting is the mismatch between excellent training performance and poor performance on unseen data. It arises from excessive model capacity, insufficient or noisy data, and data leakage, and is diagnosed primarily by the training–validation performance gap and learning curves. Strategies to mitigate overfitting include collecting more data, using data augmentation, reducing model complexity, applying regularization (L1/L2, dropout), early stopping, ensembling, and careful cross-validation. Modern machine learning has revealed nuanced behaviors (e.g., double descent) that complicate classical intuition, making continued theoretical and empirical research into generalization a critical field.

If you want, I can:

  • Run or adapt the code examples for a specific dataset (e.g., your dataset).
  • Provide a Jupyter notebook with the polynomial and neural net demos and plots.
  • Give tailored advice — a checklist — for a particular model or dataset you're working with.