Underfitting in Machine Learning: A Comprehensive Guide

Underfitting is one of the most fundamental concepts in machine learning. It describes a model that is too simple to capture the underlying structure of the data, leading to poor performance on both the training set and unseen data. In other words, an underfitted model fails to learn enough from the training data to make accurate predictions.

This article explores underfitting in depth: its meaning, causes, how it differs from overfitting, how to detect it, and how to fix it.

---

1. What Is Underfitting?

Underfitting occurs when a machine learning model cannot adequately represent the relationship between inputs and outputs in the dataset.

A model that underfits:

• Performs poorly on the training data
• Performs poorly on test/validation data
• Has high bias
• Usually has low variance

Simple intuition

Imagine trying to fit a straight line to data that clearly follows a curved pattern. A linear model may be too limited to capture the true structure. As a result, it misses important patterns and makes inaccurate predictions. That is underfitting.

---

2. Underfitting vs. Overfitting

Underfitting is often discussed alongside overfitting.

Key difference

• Underfitting means the model has not learned enough.
• Overfitting means the model has learned too much, including noise and irrelevant details.

---

3. Why Underfitting Happens

Underfitting can occur for several reasons:

3.1 Model is too simple

The model may lack the capacity to represent the complexity of the data.

Examples:

• Linear regression for highly nonlinear data
• A shallow decision tree for a complex classification task
• Too few neurons in a neural network

3.2 Poor feature selection

The input features may not contain enough useful information.

If important variables are missing, even a powerful model may underfit.

3.3 Excessive regularization

Regularization helps prevent overfitting, but too much of it can constrain the model too severely.

Examples:

• Very large L1/L2 penalty
• Dropout rate set too high in neural networks

3.4 Insufficient training

The model may not have been trained long enough or may not have converged.

Examples:

• Too few epochs
• Learning rate too low or too high
• Optimization stopped too early

3.5 Improper data preprocessing

If the data is noisy, poorly scaled, or incorrectly encoded, the model may struggle to learn meaningful patterns.

3.6 Wrong algorithm choice

Some algorithms are inherently better suited to certain types of problems.

For example:

• Linear models may underfit highly complex image or language tasks
• A simple probabilistic classifier may not capture interactions among variables

---

4. Bias, Variance, and Underfitting

Underfitting is closely related to the bias-variance tradeoff.

Bias

Bias is the error introduced by assuming a simplified model structure.

• High bias → model makes strong assumptions
• Underfitting is usually caused by high bias

Variance

Variance is the sensitivity of the model to changes in the training data.

• Low variance → model predictions do not change much across datasets
• Underfitted models typically have low variance because they are too rigid

Relationship

• Underfitting = high bias, low variance
• Overfitting = low bias, high variance

---

5. Signs of Underfitting

You may suspect underfitting if:

• Training accuracy is low
• Validation accuracy is also low
• Training loss remains high
• Test loss is similar to training loss, but both are unsatisfactory
• The model’s predictions are overly simplistic
• Residuals show obvious patterns instead of random noise

Example symptom

If a regression model consistently misses trends and produces predictions that are too flat or too linear, it may be underfitting.

---

6. How to Detect Underfitting

6.1 Compare training and validation performance

If both are poor, the model likely underfits.

6.2 Plot learning curves

Learning curves show performance over training iterations or dataset size.

Typical underfitting pattern

• Training error is high
• Validation error is high
• The two curves are close together

This suggests the model cannot fit the data well, even on the training set.

6.3 Examine residuals

In regression, residuals should look random if the model is good.

If residuals show:

• Curvature
• Trends
• Systematic structure

then the model may be too simple.

6.4 Compare against a baseline

If a simple baseline performs similarly to your model, the model may not be learning enough.

---

7. Practical Examples of Underfitting

Example 1: Linear regression on nonlinear data

Suppose the true relationship is quadratic:

$y = x^2 + \epsilon$

If you fit a straight line, it will not capture the curvature.

Result: underfitting.

Example 2: Shallow decision tree

A tree with only one or two splits may not separate classes well if the decision boundary is complex.

Result: high training error and poor generalization.

Example 3: Neural network with too few parameters

A network with very few hidden units may not have enough representational power to model the data.

Result: underfitting.

---

8. How to Fix Underfitting

The goal is to increase the model’s ability to learn meaningful patterns.

8.1 Use a more complex model

Increase model capacity by choosing a more expressive algorithm.

Examples:

• Use polynomial regression instead of linear regression
• Increase tree depth
• Add more hidden layers or neurons to a neural network
• Use ensemble methods like random forests or gradient boosting

8.2 Add better features

Feature engineering can help the model capture hidden structure.

Examples:

• Interaction terms
• Polynomial features
• Domain-specific transformations
• Better embeddings for categorical/text data

8.3 Reduce regularization

If regularization is too strong, relax it.

Examples:

• Lower L1/L2 penalty
• Reduce dropout
• Allow trees to grow deeper

8.4 Train longer

Increase the number of training iterations or epochs.

Also consider:

• Better learning rate scheduling
• More effective optimization algorithms
• Early stopping only when appropriate

8.5 Improve data quality

Use cleaner, more relevant data.

Actions:

• Handle missing values
• Remove mislabeled examples
• Normalize or standardize inputs
• Encode variables correctly

8.6 Tune hyperparameters

Model performance is often sensitive to settings such as:

• Learning rate
• Depth
• Number of trees
• Number of layers
• Batch size

---

9. Underfitting in Different Model Types

9.1 Linear models

Linear models underfit when the true relationship is nonlinear or highly interactive.

Fixes:

• Polynomial terms
• Interaction features
• Kernel methods

9.2 Decision trees

Trees underfit when they are too shallow.

Fixes:

• Increase max depth
• Lower minimum samples per leaf
• Use ensembles

9.3 Neural networks

Neural networks underfit when they are too small, poorly trained, or over-regularized.

Fixes:

• Increase width/depth
• Train longer
• Adjust regularization
• Improve architecture

9.4 Support vector machines

An SVM with an overly restrictive kernel or poor hyperparameters may underfit.

Fixes:

• Change kernel
• Adjust regularization parameter C
• Tune gamma for RBF kernels

---

10. Mathematical Perspective

In supervised learning, the objective is to minimize a loss function:

$\hat{f} = \arg\min_f \sum_{i=1}^n L(y_i, f(x_i))$

An underfitted model cannot find a sufficiently good function $f$ within its hypothesis space.

This may happen because:

• The hypothesis space is too limited
• Optimization is incomplete
• Constraints prevent the model from reaching a better fit

In bias-variance terms, the expected generalization error can be decomposed conceptually into:

• Bias error
• Variance error
• Irreducible noise

Underfitting corresponds primarily to high bias.

---

11. Underfitting in Real-World Machine Learning Systems

Underfitting is common in practice when teams prioritize simplicity too strongly.

Common real-world causes

• Using a baseline model and stopping too soon
• Overly aggressive regularization in production systems
• Poor feature pipelines
• Insufficient training data preprocessing
• Misaligned model choice for the problem

Example

In fraud detection, a simple logistic regression model may fail to capture complex interactions between transaction amount, time, geography, device type, and user history. A more expressive model may be needed.

---

12. How to Balance Underfitting and Overfitting

The ideal model is one that captures the true pattern without memorizing noise.

Practical strategies

• Start with a baseline model
• Monitor training and validation curves
• Increase model complexity gradually
• Use cross-validation
• Tune regularization carefully
• Select features thoughtfully
• Evaluate with appropriate metrics

General rule

If both training and validation performance are poor, increase model complexity or improve features. If training performance is high but validation performance is poor, reduce complexity or increase regularization.

---

13. Example Workflow for Diagnosing Underfitting

---

14. Underfitting and Model Selection

Choosing the right model depends on:

• Data complexity
• Dataset size
• Noise level
• Interpretability needs
• Computational constraints

Sometimes a simpler model is preferred even if it underfits slightly, especially when:

• Interpretability is critical
• Data is limited
• Errors are costly
• The baseline is “good enough”

The goal is not always maximum complexity, but the best tradeoff between accuracy, robustness, and practicality.

---

15. Key Takeaways

• Underfitting happens when a model is too simple to learn the data’s structure.
• It leads to poor performance on both training and test data.
• It is typically associated with high bias and low variance.
• Common causes include:
  • overly simple models
  • weak features
  • too much regularization
  • insufficient training
• Fixes include:
  • increasing model complexity
  • improving features
  • reducing regularization
  • training longer
  • tuning hyperparameters

---

16. Conclusion

Underfitting is a foundational concept in machine learning because it represents a failure to learn enough from the data. It is often easier to detect than overfitting because it shows up as poor performance across the board. The key to addressing underfitting is to increase the model’s expressiveness, improve the data and features, and ensure the training process is adequate.

A well-designed machine learning system aims to strike the right balance: complex enough to learn meaningful patterns, but simple enough to generalize well.

If you'd like, I can also provide:

1. a visual explanation of underfitting vs overfitting,
2. Python examples, or
3. a short exam-style answer on underfitting.