What is data preprocessing in machine learning?

May 19, 2026··
12 min read
Open full tree
What is data preprocessing inmachine learning?Introduction to Data PreprocessingHandling Missing DataData Normalization and ScalingEncoding Categorical VariablesFeature Engineering TechniquesData Splitting Strategies

What is Data Preprocessing in Machine Learning?

Abstract
Data preprocessing is the set of techniques and operations applied to raw data to make it suitable for machine learning (ML). It transforms imperfect, heterogeneous, and often noisy inputs into clean, consistent, and informative representations that models can learn from effectively. Good preprocessing improves model accuracy, robustness, interpretability, and makes training more stable and computationally efficient. This article provides a deep, end-to-end survey: definitions, history, theoretical foundations, concrete techniques, code examples, best practices, pitfalls, tools, state of the art, and future directions.

Table of contents

  • Overview and historical context
  • Definitions and objectives
  • Why preprocessing matters (intuitions and examples)
  • Key preprocessing tasks and techniques
    • Exploratory data analysis (EDA)
    • Data cleaning
    • Missing value handling
    • Outlier detection and treatment
    • Encoding categorical variables
    • Scaling and normalization
    • Transformations (log, Box–Cox, power transforms)
    • Feature engineering and creation
    • Feature selection
    • Dimensionality reduction
    • Imbalanced data handling
    • Time-series-specific preprocessing
    • Text preprocessing
    • Image preprocessing and augmentation
  • Theoretical foundations (statistics, linear algebra, information theory)
  • Practical pipelines and code examples (scikit-learn, imbalanced-learn)
  • Best practices and checklist
  • Common pitfalls and how to avoid them
  • Tools and infrastructure (feature stores, pipelines, AutoML)
  • Current research directions and future implications
  • Summary and recommended reading

Overview and historical context

Early ML efforts (pre-2000s) placed heavy emphasis on feature engineering because models were relatively simple and data-cleaning technology sparse. As datasets grew and complex models (e.g., deep learning) matured, preprocessing remained essential but shifted: deep nets ingest raw modalities like pixels and raw text, but still need normalization, augmentation, tokenization and careful handling of labels and metadata.

In modern production ML, preprocessing is a core discipline: data pipelines, feature stores, and reproducible preprocessing logic power robust systems. Research on automated preprocessing (AutoML), fairness-aware transformations, and privacy-preserving preprocessing (differential privacy) reflects ongoing evolution.

Definitions and objectives

Data preprocessing in ML: the set of operations that convert raw inputs into forms suitable for model training and inference. Objectives include:

  • Remove or correct faulty data (cleaning).
  • Convert data types/representations so models can use them (encoding, normalization).
  • Improve signal-to-noise ratio (filtering, outlier removal).
  • Reduce dimensionality and redundant features (feature selection, PCA).
  • Create new informative features (feature engineering).
  • Make datasets balanced/representative (resampling and weighting).
  • Ensure preprocessing is reproducible, non-leaking, and applicable in production.

Why preprocessing matters (intuition and examples)

  • Many algorithms assume scaled input (K-means, K-NN, SVM). Without scaling, features with larger numeric ranges dominate.
  • Missing values break algorithms or lead to biased estimates.
  • Categorical variables must be encoded numerically; naive mapping can impose spurious ordinal relationships.
  • Unaddressed class imbalance causes models to predict majority classes always.
  • Irrelevant or noisy features increase variance and reduce generalization.
  • Feature engineering (interaction terms, date/time decomposition) can create simple signals that dramatically improve performance.

Example: Predicting house prices — adding engineered features such as "age of house", "rooms per area", or "distance to city center", or log-transforming prices, often yields much better models than feeding raw columns only.

Key preprocessing tasks and techniques

Below are the primary tasks you'll encounter, with practical notes.

1) Exploratory Data Analysis (EDA)

  • Summary stats (mean, median, std), distributions, missingness maps.
  • Visualizations: histograms, boxplots, pairplots, correlation heatmaps.
  • Purpose: detect anomalies, relationships, non-linearities, and modeling signals.

2) Data cleaning

  • Fix incorrect types (strings vs numerics).
  • Remove duplicates.
  • Standardize text (case, spacing).
  • Remove or annotate corrupt records.

3) Handling missing values

  • Strategies:
    • Deletion: drop rows or columns (only safe if missingness is small and MCAR).
    • Imputation: mean/median/mode, forward/backfill (time series), model-based (KNN, MICE).
    • Flags: add binary indicators signaling missingness (captures informative missingness).
  • Pattern of missingness is important: MCAR (missing completely at random), MAR (at random), MNAR (not at random).

4) Outlier detection and treatment

  • Methods: z-score, IQR rule, robust methods (median absolute deviation), clustering, isolation forest.
  • Options: clip/truncate, transform, separate modeling of outliers, or remove if erroneous.

5) Encoding categorical variables

  • One-hot / dummy encoding (suitable for low-cardinality nominal features).
  • Ordinal encoding (only when order is meaningful).
  • Target encoding / mean encoding (useful for high-cardinality but must avoid leakage via CV).
  • Hashing trick (scales to very high cardinality).
  • Embeddings (learned as part of a model, especially in deep learning).

Important: Use cross-validated or within-fold encoding to prevent target leakage when using target-based encodings.

6) Scaling and normalization

  • StandardScaler (z = (x - mean) / std): centers and scales.
  • MinMaxScaler (scales to [0,1]): preserves shape but compresses ranges.
  • RobustScaler (uses median and IQR): robust to outliers.
  • L2-normalization (makes feature vectors unit-norm): for text or where direction matters.
  • When to scale: before distance-based models, before regularized linear models. Tree-based models are invariant to monotonic scaling but still may benefit from scaling with mixed feature types in some workflows.

7) Transformations

  • Log transform (handles skewness, positive-valued features).
  • Box–Cox and Yeo–Johnson (family of power transforms).
  • Rank or quantile transforms (make distribution close to normal).
  • Goal: stabilize variance and improve linearity between features and target.

8) Feature engineering

  • Combining features (ratios, differences, interactions).
  • Domain-specific features: e.g., time-of-day cycles, moving averages in time-series, image gradients.
  • Automated feature generation: featuretools, Deep Feature Synthesis.

9) Feature selection

  • Filter methods: correlation, mutual information, univariate statistical tests.
  • Wrapper methods: recursive feature elimination (RFE), forward/backward selection.
  • Embedded methods: regularization (L1), tree-based feature importances.
  • Goal: reduce dimensionality, remove collinear/uninformative variables, improve generalization.

10) Dimensionality reduction

  • PCA (Principal Component Analysis): orthogonal linear projections maximizing variance.
  • SVD (used in latent semantic analysis).
  • t-SNE, UMAP (non-linear, for visualization).
  • Autoencoders (learned non-linear compressions).
  • Use to reduce noise and storage, or for visualization.

11) Handling imbalanced data

  • Resampling: undersampling majority, oversampling minority (duplicative).
  • Synthetic sampling: SMOTE, ADASYN (create synthetic minority samples).
  • Cost-sensitive learning: class weights in loss functions.
  • Evaluation metrics: use precision, recall, F1, AUC, PR curves instead of only accuracy.

12) Time series preprocessing

  • Stationarity transforms: differencing, seasonal adjustment.
  • Lag creation, rolling windows (mean, std).
  • Time-based splitting for CV: avoid shuffling across time.
  • Handling timestamps: decompose into cyclical features (sin/cos for hour/day).

13) Text preprocessing

  • Tokenization, lowercasing, stop-word removal (problem-dependent).
  • Stemming/lemmatization, n-grams.
  • Vectorization: CountVectorizer, TF-IDF, word embeddings (Word2Vec, GloVe), contextual embeddings (BERT).
  • Handling OOV (out-of-vocab): hashing, subword tokenization.

14) Image preprocessing and augmentation

  • Resizing, center-cropping, color normalization (per-channel mean/std), scaling to [0,1].
  • Data augmentation: flips, rotations, crops, color jitter, mixup, random erasing.
  • Whitening or standardization per ImageNet statistics when using pretrained models.

Theoretical foundations

Understanding the math/statistics behind preprocessing helps choose the right methods.

  • Statistical assumptions: Many models assume IID data, linearity, homoscedasticity, or Gaussianity. Preprocessing attempts to make data align with model assumptions (e.g., log transform for heteroscedastic errors).
  • Scaling and distance measures: Distance-based methods (K-NN, K-means, SVM with RBF) are sensitive to feature scale; features with larger variance dominate Euclidean distance. Standardization removes scale differences.
  • PCA: PCA finds eigenvectors of the covariance matrix (or SVD of centered data) and projects data onto principal components. It maximizes variance captured and yields orthogonal basis.
  • Regularization: L1 (Lasso) induces sparsity, L2 penalizes large weights. Preprocessing (standardization) ensures regularization penalizes features equitably.
  • Information-theoretic measures: Mutual information can rank features by information about the target.
  • Bias–variance interplay: More preprocessing and feature engineering can decrease bias (help model capture signal) but can increase variance if overfitting to idiosyncrasies of data.

Practical pipelines and code examples

Below are concrete code examples using Python, scikit-learn, and imbalanced-learn. These show best practices: using Pipeline and ColumnTransformer to avoid leakage and to make transformations reproducible.

  1. Tabular preprocessing pipeline (numeric + categorical)
Python
1import pandas as pd 2from sklearn.compose import ColumnTransformer 3from sklearn.pipeline import Pipeline 4from sklearn.impute import SimpleImputer 5from sklearn.preprocessing import StandardScaler, OneHotEncoder 6from sklearn.ensemble import RandomForestClassifier 7from sklearn.model_selection import cross_val_score 8 9# Example dataset 10df = pd.read_csv("train.csv") # replace with your file 11X = df.drop("target", axis=1) 12y = df["target"] 13 14numeric_cols = X.select_dtypes(include=["int64", "float64"]).columns.tolist() 15categorical_cols = X.select_dtypes(include=["object", "category"]).columns.tolist() 16 17numeric_transformer = Pipeline([ 18 ("imputer", SimpleImputer(strategy="median")), 19 ("scaler", StandardScaler()) 20]) 21 22categorical_transformer = Pipeline([ 23 ("imputer", SimpleImputer(strategy="most_frequent")), 24 ("onehot", OneHotEncoder(handle_unknown="ignore")) 25]) 26 27preprocessor = ColumnTransformer([ 28 ("num", numeric_transformer, numeric_cols), 29 ("cat", categorical_transformer, categorical_cols) 30]) 31 32clf = Pipeline([ 33 ("preproc", preprocessor), 34 ("model", RandomForestClassifier(n_estimators=100, random_state=42)) 35]) 36 37scores = cross_val_score(clf, X, y, cv=5, scoring="roc_auc") 38print("AUC scores:", scores, "mean:", scores.mean())
  1. Avoiding target leakage with target-encoding (use CV)
Python
1# Use category_encoders with cross-validation to prevent leakage 2from sklearn.model_selection import KFold 3import category_encoders as ce 4import numpy as np 5 6kf = KFold(n_splits=5, shuffle=True, random_state=0) 7oof = np.zeros(len(X)) 8 9for tr_idx, val_idx in kf.split(X): 10 tr_X, tr_y = X.iloc[tr_idx], y.iloc[tr_idx] 11 val_X = X.iloc[val_idx] 12 te = ce.TargetEncoder(cols=["high_card_col"]) 13 te.fit(tr_X, tr_y) 14 oof[val_idx] = te.transform(val_X)["high_card_col"].values 15# Now 'oof' contains properly cross-validated target encoding
  1. Handling imbalanced data with SMOTE in pipeline
Python
1from imblearn.pipeline import Pipeline as ImbPipeline 2from imblearn.over_sampling import SMOTE 3from sklearn.linear_model import LogisticRegression 4 5pipeline = ImbPipeline([ 6 ("preproc", preprocessor), 7 ("smote", SMOTE(random_state=42)), 8 ("clf", LogisticRegression(max_iter=1000)) 9]) 10 11scores = cross_val_score(pipeline, X, y, cv=5, scoring="f1") 12print("F1 scores:", scores, "mean:", scores.mean())
  1. Text pipeline with TF-IDF and classifier
Python
1from sklearn.feature_extraction.text import TfidfVectorizer 2from sklearn.linear_model import LogisticRegression 3from sklearn.pipeline import make_pipeline 4 5text_pipeline = make_pipeline( 6 TfidfVectorizer(max_features=20000, ngram_range=(1,2)), 7 LogisticRegression(class_weight='balanced', max_iter=1000) 8) 9 10scores = cross_val_score(text_pipeline, df["text"], df["label"], cv=5, scoring="f1") 11print("Text F1 mean:", scores.mean())
  1. PCA for dimensionality reduction before modeling
Python
1from sklearn.decomposition import PCA 2from sklearn.pipeline import Pipeline 3from sklearn.svm import SVC 4 5pca_pipe = Pipeline([ 6 ("imputer", SimpleImputer(strategy="median")), 7 ("scaler", StandardScaler()), 8 ("pca", PCA(n_components=20)), 9 ("svc", SVC()) 10])

Best practices and checklist

  • Always split data (train/validation/test) before any fit-based preprocessing to avoid leakage.
  • Prefer Pipeline and ColumnTransformer (scikit-learn) so the same transformations apply in production.
  • Use cross-validation for parameter selection, and ensure encoding/imputation is fit inside folds.
  • Log and version preprocessing code; keep transformations deterministic (set random_state where applicable).
  • Consider adding missing-value indicators when missingness could be informative.
  • Use domain knowledge: not everything should be automated—domain features often matter most.
  • Monitor data drift and model performance in production; adapt preprocessing when input distributions change.
  • For high-cardinality categoricals, consider hashing or target encoding (careful with leakage).
  • Use robust scalers when outliers are expected.
  • For time-series, use time-aware splitting and avoid shuffling.

Common pitfalls and how to avoid them

  • Data leakage: fitting imputers, scalers, or encoders on full dataset before splitting. Fix: use pipeline and split first.
  • Target leakage: using features derived from future info or from the target. Fix: feature audit, temporal splitting.
  • Improper handling of categorical cardinality causing explosion of features with one-hot encoding. Fix: use hashing or embedding or limit cardinality.
  • Overfitting in feature selection: selecting features on full dataset then testing on the same set. Fix: perform selection within each cross-validation fold.
  • Ignoring distribution shifts: models trained on one distribution may fail when input distribution changes. Fix: monitoring, recalibration, retraining strategies.
  • Inadequate evaluation metrics for imbalanced problems: accuracy is misleading. Use precision/recall/F1/AUC-PR.

Tools and infrastructure

  • Libraries: pandas, numpy, scikit-learn, imbalanced-learn, category_encoders, featuretools, xgboost/lightgbm/catboost, spaCy, NLTK, Hugging Face Transformers, albumentations (images).
  • Production: MLflow, Airflow, Kubeflow, TFX (TensorFlow Extended) for pipelines.
  • Feature stores: Feast, Tecton (for maintaining consistent feature definitions between training and serving).
  • AutoML: Auto-sklearn, H2O AutoML, Google AutoML, AutoGluon — these often include automated preprocessing steps.

Current research directions and future implications

  • Auto-preprocessing and AutoML: automated selection of imputations, encodings, and transformations.
  • Differentiable preprocessing: making preprocessing steps part of end-to-end learnable pipelines (e.g., learned imputations, differentiable tokenizers).
  • Causally-aware preprocessing: avoiding spurious correlations and ensuring robustness via causal features.
  • Fairness-aware preprocessing: transformations that reduce bias (reweighing, disparate impact removal).
  • Privacy-preserving preprocessing: differential privacy mechanisms applied during preproc to protect sensitive info.
  • Synthetic data and generative methods to augment scarce or privacy-sensitive datasets.
  • Feature stores and standardized feature contracts across organizations to reduce errors and drift.

Examples across domains

  • Tabular (banking, insurance): imputation, target encoding of categorical features, robust scaling, feature crosses.
  • NLP: tokenization, subword vocabularies (BPE), contextual embeddings, sequence truncation/padding.
  • Computer vision: resizing, normalization with dataset means, color augmentation, random crops, use of pretrained normalization stats.
  • Time series (IoT, finance): timestamp features, lag features, rolling aggregates, differencing to remove trend.
  • Healthcare: careful missingness handling (missingness informative), privacy-preserving aggregation, clinical code embeddings.

Monitoring and maintenance in production

  • Continuously monitor feature distributions (mean/std, missingness, cardinality) and model performance.
  • Detect drift and set alerts for retraining triggers.
  • Maintain atomic, versioned preprocessing code and store transformation metadata (scalers’ means/stds, encoders’ vocabularies).
  • Ensure online and offline preprocessing match exactly — use shared libraries or feature stores.

Summary

Data preprocessing is not ancillary — it is central to successful ML. It encompasses cleaning, transforming, encoding, augmenting, and selecting features so models learn robust, generalizable patterns. Good preprocessing avoids leakage, harmonizes modalities, addresses distributional issues, and incorporates domain insight. Use reproducible pipelines, consider the statistical assumptions behind methods, and monitor deployed systems for drift. Advances in automation, privacy, and fairness will continue shaping preprocessing practice.

Recommended reading and resources

  • "An Introduction to Statistical Learning" (James, Witten, Hastie, Tibshirani) — chapters on preprocessing and linear models.
  • scikit-learn documentation: Pipelines, ColumnTransformer, preprocessing.
  • Featuretools (feature engineering automation): https://www.featuretools.com/
  • TFX (TensorFlow Extended) and Feast for production feature management.
  • Blog posts and papers on SMOTE, target encoding, and data leakage.

If you’d like, I can:

  • Walk through a tailored preprocessing pipeline for your dataset (provide a sample),
  • Help convert an ad-hoc script to a production-ready scikit-learn pipeline,
  • Demonstrate cross-validated target encoding or SMOTE usage without leakage.