Common data problems in machine learning

May 19, 2026··
14 min read
Open full tree
Common data problems in machinelearningData Quality IssuesData Preparation TechniquesData Imbalance ProblemsData Integration Challenges

Common Data Problems in Machine Learning — A Deep Dive

Machine learning (ML) models are only as good as the data they are trained on. Data issues are by far the dominant cause of poor model performance, unexpected behavior, and deployment failures. This article gives a comprehensive, practical, and theoretically informed overview of the most common data problems in machine learning, how they arise, how to detect them, workarounds and fixes, trade-offs, tools, and future directions.

Contents

  • Introduction and historical context
  • Theoretical foundations (i.i.d., generalization, bias-variance, sample complexity)
  • Catalog of common data problems
    • Missing data
    • Noisy labels / label errors
    • Class imbalance
    • Data leakage
    • Distribution shift: covariate shift, prior shift, concept drift
    • Outliers and anomalies
    • Duplicate and near-duplicate data
    • Sampling bias and selection bias
    • High dimensionality and curse of dimensionality
    • Multicollinearity and redundant features
    • Inconsistent formatting and feature engineering errors
    • Temporal and spatial dependencies (time-series-specific issues)
    • Batch effects and vendor/site effects
    • Measurement error and sensor drift
    • Privacy constraints and small data
    • Adversarial and poisoned data
  • Detection techniques and diagnostic workflows
  • Mitigation strategies and code examples
  • Tooling and best practices (data-centric ML, MLOps)
  • Case studies and examples
  • Future directions and research frontiers
  • Practical checklist
  • Conclusion and references

Introduction and historical context

Early statistical modeling and classical econometrics emphasized careful data collection and diagnostic testing (residual analysis, specification tests). As ML scaled to large datasets, the emphasis shifted to model complexity and architecture (deep networks, ensembles), sometimes deemphasizing the quality and representativeness of input data. In recent years, the community has moved back toward a data-centric view: practitioners recognize that improving data quality, labels, and coverage often yields bigger gains than tweaking models.

Prominent recent themes:

  • Data-centric AI (focus on improving datasets)
  • MLOps & continuous monitoring for data drift
  • Synthetic data, privacy-preserving training, and domain adaptation
  • Tools for data validation (Great Expectations, Deequ, TensorFlow Data Validation)

Theoretical foundations

Understanding why data problems matter requires a few theoretical ideas.

  • i.i.d. assumption: Many models assume training and test samples are independent and identically distributed. Violations (distribution shift, dependence) break guarantees.
  • Generalization: Learning theory (VC dimension, Rademacher complexity) shows that sample complexity depends on hypothesis class complexity and data distribution. Biased or insufficient data increase expected generalization error.
  • Bias–variance tradeoff: Noisy or mislabelled data increase irreducible error and can increase variance of estimators. Overfitting amplifies the impact of poor data.
  • Causal vs. associational modeling: Confounders and selection bias can produce spurious correlations. Causal frameworks (do-calculus, potential outcomes) help identify when associations will not generalize under interventions.

Practical consequence: Many mitigation strategies rely on either improving data (collect more, re-label, balance) or on methods robust to data issues (regularization, robust loss, domain adaptation).

Catalog of common data problems

Below we detail each problem with causes, impacts, detection methods, and mitigation strategies.

1) Missing data

  • What: Features or labels are missing (NaN, empty strings).
  • Why: Sensor failures, privacy redaction, merging heterogeneous sources, annotation omissions.
  • Types:
    • MCAR (Missing Completely At Random): missingness independent of observed/unobserved data.
    • MAR (Missing At Random): missingness depends only on observed variables.
    • MNAR (Missing Not At Random): missingness depends on the unobserved value itself.
  • Impact: Bias if handled incorrectly; loss of sample size; broken pipelines.
  • Detection: Summary of null counts, patterns, correlation between missingness and Y.
  • Mitigation:
    • Simple: drop rows/columns (only if data loss small and MCAR).
    • Imputation: mean/median/mode, KNN, iterative imputation (MICE), model-based imputation.
    • Feature engineering: add indicators for missingness.
    • Domain-specific: treat missing as informative category.
  • Code snippet (pandas + IterativeImputer):
Python
1import pandas as pd 2from sklearn.experimental import enable_iterative_imputer 3from sklearn.impute import IterativeImputer 4df = pd.read_csv("data.csv") 5num_cols = df.select_dtypes(include="number").columns 6imp = IterativeImputer(random_state=0) 7df[num_cols] = imp.fit_transform(df[num_cols]) 8df["was_missing_feat1"] = df["feat1"].isna().astype(int) # indicator

2) Noisy or incorrect labels

  • What: Wrong class labels or regression targets, low inter-annotator agreement, systematic labeling errors.
  • Why: Human errors, ambiguous items, label distribution shift over time, incorrect mappings.
  • Impact: Limits upper bound on accuracy; models can learn wrong decision boundaries; calibration suffers.
  • Detection:
    • Confusion across annotators (kappa), label entropy.
    • Find instances where model strongly disagrees with label.
    • Use agreement heuristics and cleanlab to estimate label noise.
  • Mitigation:
    • Re-annotate targeted samples (active learning).
    • Use robust loss functions (e.g., label smoothing, noise-robust losses).
    • Probabilistic / soft labels reflecting annotator disagreement.
    • Model-based methods: iterative relabeling, meta-learning for label noise.
  • Tools: Cleanlab, CrowdTruth.

3) Class imbalance

  • What: One or a few classes are much rarer than others.
  • Why: Natural rarity (fraud, disease), sampling procedure.
  • Impact: Poor minority class performance; accuracy paradox (high accuracy by predicting majority).
  • Detection: Class frequency table, per-class metrics (precision, recall, F1).
  • Mitigation:
    • Resampling: undersampling majority, oversampling minority (SMOTE, ADASYN).
    • Cost-sensitive learning: class weights in loss.
    • Use proper metrics: ROC-AUC, PR-AUC, per-class recall, F1.
    • Algorithm choices: tree-based models, anomaly detection for extreme imbalance.
  • Code example (SMOTE with imbalanced-learn):
Python
from imblearn.over_sampling import SMOTE X_res, y_res = SMOTE(random_state=42).fit_resample(X_train, y_train)

4) Data leakage

  • What: Information from outside the training dataset or from the future leaks into training so model sees information it would not have at inference.
  • Examples: including target-derived features, improper scaling before split, using future timestamps in training.
  • Why: Incorrect pipeline ordering, naive feature engineering, using aggregated labels.
  • Impact: Over-optimistic validation, catastrophic failure in production.
  • Detection: Unrealistic performance gap between validation and test/production; features with extremely high importance that are suspicious.
  • Mitigation:
    • Strict separation of training/validation/test datasets.
    • Use pipeline primitives (scikit-learn Pipeline) so transformations are fit only on training folds.
    • Review feature provenance; remove features that are temporally downstream of the label.
  • Code example (correct pipeline):
Python
1from sklearn.pipeline import Pipeline 2from sklearn.impute import SimpleImputer 3from sklearn.preprocessing import StandardScaler 4clf_pipeline = Pipeline([ 5 ('imputer', SimpleImputer()), 6 ('scaler', StandardScaler()), 7 ('clf', LogisticRegression()) 8]) 9clf_pipeline.fit(X_train, y_train) # safe

5) Distribution shift (covariate shift, prior shift, concept drift)

  • What: Training and test distributions differ.
    • Covariate shift: P(X) changes, P(Y|X) stable.
    • Prior/class-probability shift: P(Y) changes.
    • Concept drift: P(Y|X) changes (most pernicious).
  • Why: Nonstationary environments (finance, user behavior), different data collection.
  • Impact: Model performance degrades over time or on new domains.
  • Detection: Performance monitoring, two-sample tests (KS, MMD), drift detection algorithms.
  • Mitigation:
    • Domain adaptation (importance weighting for covariate shift).
    • Continual learning and online updating.
    • Retraining schedules, monitoring, A/B tests.
    • Use invariant features or causal features.
  • Importance weighting example (simple):
Python
# Fit propensity model p(train|x) to reweight # More advanced: kernel mean matching, domain adversarial networks

6) Outliers and anomalies

  • What: Extreme observations that do not conform to expected patterns.
  • Why: Rare events, measurement errors, fraud.
  • Impact: Distorts model estimates (esp. mean-based), harms some algorithms.
  • Detection: Boxplots, z-score, IQR rule, isolation forest, robust PCA.
  • Mitigation:
    • Remove or cap (winsorize) if erroneous.
    • Robust models (quantile regression, tree-based models).
    • Treat as separate class (anomaly detection).
  • Code example (IsolationForest):
Python
from sklearn.ensemble import IsolationForest iso = IsolationForest(contamination=0.01) outlier_labels = iso.fit_predict(X)

7) Duplicate and near-duplicate data

  • What: Duplicate rows or very similar items (e.g., scraping duplicates).
  • Why: Aggregation errors, data scraping, multiple observations of same entity.
  • Impact: Inflated evaluation metrics if duplicates appear across train/test splits; biased learning.
  • Detection: Hashing, fuzzy matching, clustering on embeddings.
  • Mitigation:
    • Remove duplicates.
    • When splitting, group by entity id to keep duplicates in same fold.
  • Code snippet:
Python
df.drop_duplicates(subset=["text", "user_id"], inplace=True)

8) Sampling bias and selection bias

  • What: Training data is not representative of the target population due to how it was collected.
  • Examples: Survey non-response, web logs only for active users.
  • Impact: Models that do not generalize, fairness issues.
  • Detection: Compare sample demographics to population, domain knowledge.
  • Mitigation:
    • Reweighting (inverse probability weighting), stratified sampling, targeted data collection.
    • Use causal models to adjust for selection mechanisms.

9) High dimensionality and the curse of dimensionality

  • What: Too many features relative to samples, sparse signal.
  • Impact: Overfitting, high variance, expensive compute.
  • Mitigation:
    • Feature selection, regularization (L1, elastic net).
    • Dimensionality reduction (PCA, autoencoders).
    • Use domain knowledge to engineer informative features.

10) Multicollinearity and redundant features

  • What: Strong correlations between features.
  • Impact: Instability in coefficient estimates (linear models), reduced interpretability.
  • Detection: Correlation matrix, Variance Inflation Factor (VIF).
  • Mitigation: Drop or combine correlated features, use regularized models.
  • VIF calculation snippet:
Python
from statsmodels.stats.outliers_influence import variance_inflation_factor vif = {col: variance_inflation_factor(X.values, i) for i,col in enumerate(X.columns)}

11) Inconsistent formatting, units, and preprocessing errors

  • What: Features with mixed units (meters vs feet), inconsistent categorical encodings, date parsing errors.
  • Impact: Garbage-in results, silent failures.
  • Detection: Unit checks, value ranges, domain validation rules.
  • Mitigation: Standardize units/formats; use data validation frameworks.

12) Temporal and spatial dependencies (time-series-specific issues)

  • What: Autocorrelation, seasonality, nonstationarity; leakage through improper time splits.
  • Impact: Overoptimistic evaluation; step-change failures on real deployment.
  • Detection: ACF/PACF plots, rolling performance, time-split validation.
  • Mitigation:
    • Use time-based cross-validation (walk-forward validation).
    • Feature lagging carefully; avoid future information.
  • Example: use GroupKFold or TimeSeriesSplit rather than random split.

13) Batch effects and site/vendor effects

  • What: Variation from batch processing, vendor differences (common in genomics, imaging).
  • Impact: Confounding where batch correlates with label.
  • Detection: PCA colored by batch, ANOVA on features by batch.
  • Mitigation:
    • Batch correction techniques (ComBat, harmonization).
    • Randomize batches across classes during data collection.

14) Measurement error and sensor drift

  • What: Systematic or random error in measurement instruments.
  • Impact: Bias and degradation over time; some features lose predictive power.
  • Mitigation:
    • Calibration, sensor redundancy, drift detection and recalibration.
    • Use robust estimation methods.

15) Privacy constraints and small data

  • What: Data cannot be shared or is intentionally limited (privacy regulations).
  • Impact: Smaller effective datasets, limited feature availability.
  • Mitigation:
    • Federated learning, synthetic data generation, differential privacy (trade-offs).
    • Transfer learning and pretraining.

16) Adversarial and data poisoning attacks

  • What: Maliciously crafted inputs or training data to degrade model performance.
  • Impact: Security failures in deployed models.
  • Mitigation:
    • Data provenance checks, robust training, adversarial training, anomaly detection on training set.

Detection techniques and diagnostic workflows

A structured workflow reduces risk of missing issues:

  1. Data inventory and provenance: catalog sources, collection methods, update frequencies.
  2. Summary statistics:
    • Value distributions, missingness, unique values, cardinality.
  3. Visual EDA:
    • Histograms, boxplots, correlation heatmaps, PCA/TSNE colored by labels/batches.
  4. Split aware checks:
    • Compare train/validation/test distributions; group-aware split checks.
  5. Label analysis:
    • Confusion matrices, annotator agreement stats, examine high-loss samples.
  6. Time-aware diagnostics:
    • Rolling metrics, drift detection.
  7. Two-sample tests:
    • KS-test for univariate features, MMD/KL for multivariate comparisons.
  8. Automated checks:
    • Great Expectations, TensorFlow Data Validation, Evidently for drift monitoring.

Example: two-sample drift detection in Python (univariate KS):

Python
from scipy.stats import ks_2samp stat, p = ks_2samp(train["age"].dropna(), prod["age"].dropna())

Mitigation strategies — practical examples and code

Below are practical solutions with code snippets.

  1. Handling missing data
  • Use indicators and model-based imputation:
Python
1from sklearn.impute import SimpleImputer 2imp = SimpleImputer(strategy="median") 3X_train_imp = imp.fit_transform(X_train) 4X_test_imp = imp.transform(X_test)
  1. Robust training with noisy labels
  • Use cross-validation disagreements and re-label:
Python
# Pseudocode: select top k samples with highest loss across CV to re-annotate
  • Use Cleanlab to estimate and correct label noise.
  1. Class imbalance
  • Combine class weights with stratified sampling:
Python
from sklearn.linear_model import LogisticRegression clf = LogisticRegression(class_weight="balanced")
  1. Distribution shift mitigation via importance weighting
  • Compute density ratio p_test(x)/p_train(x) using a classifier:
Python
# Train a classifier to distinguish train vs test; use predicted odds as weights
  1. Preventing leakage
  • Ensure entire pipeline is nested inside cross-validation:
Python
from sklearn.model_selection import cross_val_score cross_val_score(clf_pipeline, X, y, cv=5)
  1. Temporal validation
Python
1from sklearn.model_selection import TimeSeriesSplit 2tscv = TimeSeriesSplit(n_splits=5) 3for train_idx, test_idx in tscv.split(X): 4 # fit only on train indices
  1. High-dimensionality
  • Use feature selection and regularization:
Python
1from sklearn.feature_selection import SelectFromModel 2from sklearn.linear_model import LassoCV 3sfm = SelectFromModel(LassoCV()) 4X_small = sfm.fit_transform(X, y)
  1. Batch effect correction (example concept)
  • ComBat harmonization in neuroimaging/genomics.

Tooling and best practices (data-centric ML and MLOps)

  • Data validation: Great Expectations, Deequ, TFDV, Evidently.
  • Label quality: Cleanlab, Prodigy (annotation tooling), Label Studio.
  • Drift monitoring: Evidently, Fiddler, WhyLabs, Seldon Analytics.
  • Pipeline automation: TFX, Kubeflow, MLflow, DVC for data versioning.
  • Practices:
    • Version your datasets and transformations.
    • Maintain a data contract: expectations on ranges, types, cardinalities.
    • Implement continuous monitoring and alarms for covariate/class/performance drift.
    • Keep a small “canary” production holdout dataset or shadow evaluation.

Case studies and examples

  1. Healthcare (radiology classification)
  • Problems: label noise from inter-reader variability; small datasets; domain shift across hospitals.
  • Solutions: consensus labels, transfer learning, domain harmonization, federated learning.
  1. Fraud detection (finance)
  • Problems: extreme class imbalance, adversarial adaptation, nonstationarity.
  • Solutions: anomaly detection methods, cost-sensitive learning, frequent model updates, simulation of adversarial scenarios.
  1. NLP sentiment dataset
  • Problems: label ambiguity, duplicate scraping, distribution shift between training social media and production product reviews.
  • Solutions: active re-labeling, deduplication, fine-tune on in-domain data.
  1. Sensor IoT
  • Problems: missing data and drift from sensor degradation.
  • Solutions: imputation using temporal models, sensor redundancy, drift detectors and recalibration.

Current state and future implications

Current trends:

  • Shift toward data-centric AI: systematic dataset improvement becomes a primary lever.
  • Automation: automated data validation and drift detection integrated into MLOps.
  • Privacy and synthetic data: rising use of synthetic data and differential privacy; ongoing trade-offs between utility and privacy.
  • Domain adaptation & unsupervised transfer learning: models robust to distribution shift, self-supervised learning to leverage unlabeled in-domain data.
  • Labeling systems and active learning: more sophisticated annotation platforms, consensus-based labels, ML-assisted labeling.

Research frontiers:

  • Theoretical understanding of training under label noise for deep networks.
  • Provable domain adaptation techniques and invariant representation learning.
  • Methods for explainable dataset diagnostics that link data problems to model errors.
  • Robustness to adversarial poisoning in distributed/federated settings.

Practical checklist for dataset health

Before training a model, run this checklist:

  1. Data inventory: sources, timestamps, update cadence, ownership.
  2. Missingness check: proportion, patterns, check MCAR/MAR/MNAR possibilities.
  3. Label quality: annotate agreement, inspect ambiguous cases, measure label-classifier discordance.
  4. Duplicates: remove/aggregate duplicates; ensure grouping of duplicates when splitting.
  5. Distribution checks: feature-wise comparisons across train/val/test, two-sample tests.
  6. Temporal checks: ensure no future data leakage; use time-aware splits.
  7. Outliers: visualize and decide case-by-case whether to remove or model explicitly.
  8. Class balance & metrics: pick appropriate evaluation metrics and sampling strategies.
  9. Provenance & lineage: track transformations and versions of datasets.
  10. Monitoring plan: deploy drift detection and retraining triggers.

Conclusion

Poor or mismanaged data is a primary root cause of poor machine learning outcomes. Understanding and systematically addressing data problems—missing values, label noise, distribution shift, leakage, imbalance, and others—is essential for reliable, fair, and robust ML systems. The contemporary best practice is data-centric: measure, test, and improve the dataset along with model development; instrument continuous monitoring for drift; and bring domain expertise into data curation.

Selected references and tools

  • Tools: Great Expectations, TensorFlow Data Validation, Evidently, Cleanlab, imbalanced-learn, scikit-learn, DVC, MLflow.
  • Reading:
    • "Data-Centric AI" movement resources (Andrew Ng and community articles).
    • "The Elements of Statistical Learning" (Hastie, Tibshirani, Friedman) — general theory.
    • Papers on covariate shift, domain adaptation, label noise robustness.
    • Practical MLOps resources and documentation for data validation tools.

If you want, I can:

  • Produce a runnable notebook that demonstrates detection and fixes for a couple of these problems (missing values, class imbalance, data leakage).
  • Provide checklists or templates tailored to a specific domain (healthcare, finance, NLP).
  • Walk through a concrete dataset you provide and run diagnostics.