What is a recurrent neural network?

May 10, 2026··

15 min read

What is a Recurrent Neural Network?

A recurrent neural network (RNN) is a class of artificial neural networks designed to process sequential data by maintaining a hidden state that evolves over time. Unlike feedforward networks, RNNs have temporal dynamics: the output at each time step depends not only on the current input but also on previous inputs through the internal state. This makes them well-suited to tasks where order and context matter: language, speech, time series, music, control signals, and more.

This article provides a comprehensive, in-depth overview of RNNs: history, fundamental concepts, mathematical foundations, popular architectures (including LSTM and GRU), training methods, common problems and solutions, practical applications, code examples, current state-of-the-art and future directions, and recommended resources.

Table of contents

High-level intuition and motivation
Historical development
Formal definition and notation
Core RNN architectures
- Vanilla RNN (Elman)
- Jordan network
- Long Short-Term Memory (LSTM)
- Gated Recurrent Unit (GRU)
- Bidirectional and stacked RNNs
Training RNNs
- Forward pass equations
- Backpropagation Through Time (BPTT)
- Vanishing and exploding gradients
- Practical training techniques
Sequence modeling tasks and architectures
- Sequence classification and labeling
- Sequence generation and language modeling
- Encoder–decoder (seq2seq) and attention mechanisms
Implementation examples (PyTorch, TensorFlow/Keras)
Practical considerations and best practices
Applications and examples
Current trends and the role of transformers
Future directions and research topics
Key references and further reading

High-level intuition and motivation

Humans interpret sequential data by remembering context. Consider reading a sentence: each word is interpreted in light of preceding words. Classic feedforward networks lack persistence: they treat inputs independently. RNNs introduce memory via a hidden state vector that "remembers" information from previous time steps. The hidden state acts as a dynamic summary of past inputs and is updated recurrently as new data arrives.

Key motivations:

Modeling sequences (variable-length input and/or output)
Capturing temporal dependencies and context
Enabling online/streaming processing (stateful inference)
Parameter sharing across time steps reduces model size and helps generalization

Historical development

1980s: Early ideas of networks with feedback and temporal dynamics. Attributed pioneers include John Hopfield (associative memories), and early recurrent architectures.
1987–1990s: Elman networks (1990) and Jordan networks (1986) developed for tasks with temporal dependencies.
- Elman (1990): Simple recurrent network with context units that store previous hidden activations.
- Jordan (1986): Context units fed by previous outputs.
1990s: Backpropagation Through Time (BPTT) formalized to train recurrent nets.
1997: Hochreiter & Schmidhuber introduced Long Short-Term Memory (LSTM), addressing vanishing gradient problems and enabling learning of long-range dependencies.
2014: Cho et al. introduced Gated Recurrent Unit (GRU), a simplified gated alternative to LSTM.
2014–2020s: RNNs (LSTMs/GRUs) became standard in many sequence tasks (speech recognition, machine translation, language modeling).
2017 onwards: Transformers (attention-only architectures) disrupted the field by outperforming RNNs on many tasks. Yet RNNs remain relevant in streaming, compact models, and certain time-series contexts.

Formal definition and notation

Consider an input sequence x = (x1, x2, ..., xT), where xt ∈ R^n. Let ht ∈ R^m denote the hidden state at time t, and yt ∈ R^k denote the output at time t (optional). The recurrent update is typically:

ht = f(Whh ht-1 + Wxh xt + bh)

yt = g(Why ht + by)

Where:

Wxh: weights from input to hidden
Whh: recurrent weights (hidden to hidden)
Why: hidden to output weights
bh, by: bias vectors
f: activation function (tanh, ReLU, etc.)
g: output activation (softmax for classification, linear for regression)

Important aspects:

The same weights are applied at every time step (parameter sharing).
The initial hidden state h0 may be learned or set to zeros.
For variable-length sequences, the recurrence runs up to T.

This simple formulation is the "vanilla" or "Elman" RNN.

Core RNN architectures

1) Vanilla RNN (Elman network)

Update equations:

ht = φ(Wxh xt + Whh ht-1 + bh)
yt = ψ(Why ht + by)

Here φ is usually tanh or ReLU, and ψ depends on the task.

Strengths:

Simple, efficient for short-range dependencies.

Weaknesses:

Training over long sequences often fails due to vanishing or exploding gradients.

2) Jordan network

In a Jordan network, the context (recurrent input) is the previous output rather than previous hidden state. Less common now.

3) Long Short-Term Memory (LSTM)

LSTM introduces memory cell ct and gating mechanisms to allow gradients to flow across many time steps. LSTM's design addresses the vanishing gradient issue through the cell state and multiplicative gates.

Standard LSTM equations (one common variant):

it = σ(Wxi xt + Whi ht-1 + bi) (input gate) ft = σ(Wxf xt + Whf ht-1 + bf) (forget gate) ot = σ(Wxo xt + Who ht-1 + bo) (output gate) g t = tanh(Wxg xt + Whg ht-1 + bg) (cell candidate) ct = ft ⊙ ct-1 + it ⊙ g t (cell state update) ht = ot ⊙ tanh(ct) (hidden state / output)

Where:

σ is the sigmoid function
⊙ is element-wise multiplication
gates constrain information flow, enabling long-term storage

Advantages:

Handles long-range dependencies
Widely used in NLP, speech, time series

4) Gated Recurrent Unit (GRU)

GRU simplifies LSTM by combining gates and merging cell and hidden state:

zt = σ(Wxz xt + Whz ht-1 + bz) (update gate) rt = σ(Wxr xt + Whr ht-1 + br) (reset gate) ht~ = tanh(Wxh xt + Whh (rt ⊙ ht-1) + b) ht = (1 - zt) ⊙ ht-1 + zt ⊙ ht~

GRUs often match LSTMs in performance while being computationally cheaper.

5) Bidirectional and stacked RNNs

Bidirectional RNNs (BiRNN): Process sequence forward and backward and combine states: useful when entire sequence is available (e.g., text tagging).
Stacked (multi-layer) RNNs: Multiple recurrent layers where outputs of one layer feed the next. Improves representational capacity.

Training RNNs

Forward pass

At each time step compute hidden state and output with recurrence equations. For sequences in batches, time-major or batch-major layouts are used; sequences may be padded and masked.

Backpropagation Through Time (BPTT)

BPTT unfolds the RNN across time steps into an equivalent deep feedforward network and applies backpropagation to compute gradients. For T time steps, gradients are backpropagated through T layers.

Key issues:

BPTT across very long sequences is computationally expensive and memory intensive.
Truncated BPTT: Backpropagate gradients for a limited window (e.g., 20–50 steps), trade-off between temporal credit assignment and efficiency.

Vanishing and exploding gradients

As gradients are propagated through many time steps, repeated multiplication by weight matrices and derivatives can lead to:

Vanishing gradients: Gradients exponentially decay, making learning of long-range dependencies difficult.
Exploding gradients: Gradients grow exponentially, causing training instability.

Explanation (qualitatively): gradient ∂L/∂h_t depends on powers of Whh and derivatives of activation. If eigenvalues of Whh are less than 1, gradient decays; if greater than 1, it explodes.

Remedies:

Gated architectures (LSTM/GRU) mitigate vanishing gradients via additive cell updates and gating.
Gradient clipping (e.g., clip norm to threshold) prevents exploding gradients.
Orthogonal or unitary recurrent matrices, specialized RNN variants.
Careful initialization (e.g., orthogonal initialization).
Use of activations less prone to saturating gradients (ReLU with caution).
Layer normalization, batch normalization variants (though batch norm is trickier in recurrent settings).

Practical training techniques

Mini-batching and packing variable-length sequences
Truncated BPTT for long sequences
Teacher forcing for sequence generation (provide ground-truth previous token during training); alternatives: scheduled sampling
Regularization: dropout (variational dropout/time-step consistent dropout), weight decay
Optimizers: Adam, RMSprop, SGD with momentum
Learning rate schedules and warmup

Sequence modeling tasks and architectures

RNNs are versatile for many sequence tasks. Below are common setups and model architectures.

Sequence classification

Input: sequence, Output: single label (e.g., sentiment analysis).
Typical architecture: RNN (or BiRNN) encodes sequence; final hidden state pooled (last state or mean/max pooling) → classifier (softmax).
Loss: cross-entropy for classification.

Sequence labeling (token-level predictions)

Input: sequence, Output: label per time step (e.g., POS tagging, NER).
Architecture: BiRNN with per-step classifier, or BiRNN + CRF on top for structured outputs.

Sequence generation and language modeling

Task: predict next token given previous tokens.
Models: RNN/LSTM/GRU with softmax output across vocabulary.
Training: teacher forcing or variants. Evaluation: perplexity (exp of cross-entropy), BLEU (for translation), accuracy.

Encoder–decoder (seq2seq)

Encoder RNN reads an input sequence into a context vector (final hidden state(s)).
Decoder RNN generates output sequence conditioned on context. Without attention, compression to a single vector is limiting for long sequences.
Attention mechanisms (Bahdanau, Luong) resolve bottleneck by allowing decoder to attend to encoder states at each time step.

Sequence-to-sequence with attention

Each decoder step computes attention weights over encoder hidden states and uses weighted context as additional input; this enables long-range alignment and better performance in tasks like machine translation and summarization.

Implementation examples

Below are minimal code examples illustrating core concepts. These are simplified; production code requires padding, masking, batching, and careful training.

PyTorch: Simple RNN/LSTM for sequence classification

Python

import torch
import torch.nn as nn

class SimpleLSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes, n_layers=1, bidirectional=False):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers=n_layers, 
                            bidirectional=bidirectional, batch_first=True)
        factor = 2 if bidirectional else 1
        self.fc = nn.Linear(hidden_dim * factor, num_classes)
    
    def forward(self, x, lengths=None):
        # x: (batch, seq_len)
        emb = self.embedding(x)                # (batch, seq_len, embed_dim)
        packed = nn.utils.rnn.pack_padded_sequence(emb, lengths, batch_first=True, enforce_sorted=False) \
                 if lengths is not None else emb
        out, (hn, cn) = self.lstm(packed if lengths is not None else emb)
        # If packed, unpack:
        if lengths is not None:
            out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
        # Use last hidden state(s)
        if self.lstm.bidirectional:
            last = torch.cat([hn[-1], hn[-2]], dim=1)
        else:
            last = hn[-1]
        logits = self.fc(last)
        return logits

TensorFlow / Keras: LSTM language model (toy)

Python

import tensorflow as tf

vocab_size = 10000
embed_dim = 256
hidden_dim = 512

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embed_dim),
    tf.keras.layers.LSTM(hidden_dim, return_sequences=True),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(vocab_size))
])

# compile and train with sparse_categorical_crossentropy
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True))

Seq2seq with attention (conceptual pseudo-code):

Encoder: bidirectional LSTM → encoder outputs H = [h1...hT]
Decoder: at time t, compute attention weights αt over encoder outputs via feedforward scorer: αt = softmax(score(ht-1_decoder, H)) context = Σ αt,i * hi input to decoder = concat(embedding(y_{t-1}), context) decode one step → output distribution

Many frameworks (PyTorch, TensorFlow Addons, OpenNMT) provide building blocks.

Practical considerations and best practices

Dealing with variable-length sequences:
- Pad sequences and use masks to ignore padding during loss computation.
- Use packed sequences in PyTorch (pack_padded_sequence) to speed up computation.
Mini-batching and state management:
- For stateful RNNs across batches, ensure sequences are aligned in time; often easier to reset state per batch.
- Keep batch sizes moderate; gradient noise trade-offs.
Gradient clipping:
- Use gradient norm clipping (e.g., clip if ||g||>5).
Initialization:
- Orthogonal initialization for recurrent weights can help stabilize training.
- Bias initialization: for LSTM forget gate bias, initializing to a positive value (e.g., 1) encourages remembering initially.
Regularization:
- Dropout: apply dropout between layers or use "variational dropout" that uses the same dropout mask across time steps (e.g., nn.Dropout in PyTorch applied to inputs or outputs per time step should be consistent).
- Weight decay and early stopping.
Teacher forcing and scheduled sampling:
- Teacher forcing speeds training, but can cause exposure bias at inference. Scheduled sampling gradually replaces ground-truth prev tokens with model's samples.
Monitoring and evaluation:
- Use appropriate sequence metrics: perplexity for language models, BLEU/ROUGE for generation tasks, F1 for sequence labeling.
Efficiency:
- Use cuDNN-accelerated RNN implementations (torch.nn.LSTM with batch_first and device=GPU).
- For long sequences consider truncation, hierarchical RNNs, or convolutional/transformer alternatives.

Applications and examples

RNNs have been applied extensively:

Natural Language Processing:
- Language modeling, text generation
- Machine translation (seq2seq + attention)
- Named Entity Recognition (NER), Part-of-Speech tagging
- Question answering (as components)
Speech and audio:
- Speech recognition (ASR): RNNs/LSTMs process audio features
- Speech synthesis (e.g., Tacotron used RNNs earlier)
Time series forecasting:
- Financial forecasting, energy demand, sensor data
Signal processing:
- Anomaly detection in sequential sensor streams
Music and art generation:
- RNN-based music models generate sequences of notes
Control and robotics:
- Policy networks with memory; learning from sequential sensor inputs
Video and multimodal:
- RNNs have processed sequences of frame features for captioning and activity recognition

Examples:

The original neural machine translation systems (Sutskever et al. 2014, Bahdanau et al. 2014) used LSTM encoder-decoder with attention.
Speech recognition systems (Deep Speech, prior to transformer-based/Conformer models) used RNNs/LSTMs extensively.

Current trends and the role of transformers

In the last 6–7 years, transformer architectures (Vaswani et al., 2017) based on self-attention have largely superseded RNNs in many tasks—especially in NLP. Reasons:

Parallelizable training (no sequential recurrence), enabling faster training on GPUs/TPUs
Better modeling of long-range dependencies via attention
Scalability to very large models and pre-training (BERT, GPT, etc.)

However, RNNs remain relevant in several niches:

Streaming or online inference with low latency: transformers require attention over the full context or careful chunking; specialized streaming transformer variants exist but RNNs naturally maintain state.
Parameter/compute-efficient models: RNNs can be more compact for certain tasks.
Edge and embedded devices where memory, latency, and compute are constrained.
Specialized recurrent variants (unitary RNNs, spiking RNNs) are subjects of active research.

Hybrid approaches exist: combining convolutional front-ends, recurrent processing for short-term memory, and attention modules for longer context.

Future directions and research topics

Efficient recurrent architectures for long sequences and streaming
Unitary and orthogonal RNNs that preserve gradient norm
Spiking neural networks and biologically plausible recurrent models
Continual learning with recurrent models (memory consolidation)
Interpretability of learned recurrent states
Hybrid models: RNNs with attention, or efficient transformer recurrence
Low-power hardware accelerators optimized for recurrent operations
Improved regularization and stability techniques for extremely long-range dependencies
Differentiable memory systems and external memory augmentation (Neural Turing Machines, Memory Networks)

Mathematical appendix — gradients and vanishing/exploding

Consider simplified scalar recurrence and loss L that depends on hT. The derivative ∂L/∂Whh involves products:

∂L/∂Whh = Σ_{t=1..T} (∂L/∂hT) (Π_{k=t+1..T} f'(ak) Whh) f'(at) ht-1

Where f' are activation derivatives. The product Π_{k} Whh f'(ak) may grow or shrink exponentially with (T - t). If the spectral radius (largest absolute eigenvalue) of Whh multiplied by average |f'| is <1, gradients vanish; >1, explode.

LSTM mitigates this because the cell state update ct = ft ⊙ ct-1 + it ⊙ g t is additive, enabling gradient flow through ct across many time steps when ft ≈ 1 and it ≈ 0, preserving information.

Gradient clipping commonly implemented as: if ||g||_2 > threshold: g = g * (threshold / ||g||_2)

Checklist and practical recipes

Model choice:
- Small dataset, short dependencies: vanilla RNN or GRU
- Long-term dependencies: LSTM or GRU
- Bidirectional necessary? Use BiRNN if full sequence available
Optimizer: Adam or RMSprop to start; switch to SGD if needed
Initialization: orthogonal for recurrent weights, Xavier/Glorot for others
Learning rates: start with 1e-3 for Adam; tune
Batch size: depends on memory; larger batch sizes help parallelism but watch learning dynamics
Truncation: for long sequences, truncated BPTT windows (20–200 steps)
Evaluation: use task-appropriate metrics (perplexity, BLEU, F1)
Debugging: check gradient norms, overfitting, underfitting, and training vs validation loss

Resources and further reading

Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation.
Elman, J. L. (1990). Finding structure in time. Cognitive Science.
Graves, A. (2012). Supervised Sequence Labelling with Recurrent Neural Networks. PhD thesis (Cambridge).
Cho, K. et al. (2014). On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. (Introduced GRU-like variants)
Bahdanau, D., Cho, K., Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. (Attention)
Vaswani, A. et al. (2017). Attention Is All You Need. (Transformers)
Good tutorials:
- PyTorch and TensorFlow official RNN/LSTM tutorials
- Andrej Karpathy's "The Unreasonable Effectiveness of Recurrent Neural Networks" blog post
Libraries and frameworks:
- PyTorch (nn.RNN, nn.LSTM, nn.GRU), TensorFlow/Keras (tf.keras.layers.LSTM, GRU)
- OpenNMT, Fairseq, HuggingFace Transformers (for seq2seq and transformer baselines)

Summary

Recurrent Neural Networks are foundational neural architectures for sequential data, characterized by temporal recurrence and hidden state dynamics. While vanilla RNNs struggle with long-term dependencies due to vanishing/exploding gradients, gated architectures (LSTM and GRU) successfully mitigate these issues and powered many breakthroughs in speech and language tasks. Today, transformers challenge RNN dominance by offering greater parallelism and stronger long-range modeling, but RNNs still occupy important niches—especially in streaming, efficient, or resource-constrained environments. Understanding RNNs, their training dynamics, and practical pitfalls remains important for practitioners and researchers working with sequential data.

If you'd like, I can:

Provide a runnable PyTorch example for training an LSTM language model on a small dataset.
Walk through deriving BPTT gradients step-by-step.
Compare RNNs and transformers on specific tasks (memory/compute tradeoff).
Give a checklist to convert an RNN model to a transformer or hybrid.