Recurrent Neural Networks, Sequence Training, and Embeddings

Chapter Overview

This chapter covers the sequence-modeling lecture: recurrent neural-network architectures, backpropagation through time, vanishing and exploding gradients in the recurrent setting, gating with LSTM and GRU, truncated BPTT, 1-D convolutions for sequences, autoencoders, and embeddings.

It matters because the course treats recurrence as the natural extension of neural networks from static inputs to time-dependent or ordered inputs. The exam-aligned questions are about the recurrence equation, the specific meaning of vanishing gradient in RNNs, and the computational motivation for TBPTT.

The lecture's key conceptual shift is parameter sharing over time rather than over space. Once the same recurrent weights are reused across many time steps, memory and gradient propagation become the central issues.

Important Definitions

• Hidden state. Internal recurrent memory vector carrying information from one time step to the next.
• Unfolded RNN. Time-expanded view of a recurrent network where one recurrent cell becomes a long chain of tied copies.
• Deep RNN. Recurrent network with multiple hidden layers stacked vertically.
• Bi-directional RNN (BRNN). Sequence model using one recurrence forward in time and one backward in time.
• BPTT. Backpropagation through time: gradient computation on the unfolded recurrent graph.
• Vanishing / exploding gradient in RNNs. Product of many Jacobian factors shrinks or grows exponentially, making early-time influence too small or too unstable.
• Gradient clipping. Cap large gradients to stabilize training.
• LSTM. Gated recurrent architecture introducing a cell state and gates to mitigate structural vanishing/exploding along the memory path.
• GRU. Simpler gated recurrent unit with reset and update gates and no separate cell state.
• TBPTT. Truncated backpropagation through time, where the forward and backward horizons are limited.
• Autoencoder. Network mapping data to itself through a latent representation, often with a lower-dimensional bottleneck.
• One-hot encoding. Sparse discrete representation with one active coordinate per category.
• Embedding. Learned dense continuous representation of a discrete category.

Key Concepts and Intuition

• Recurrence means parameter sharing across time. The same recurrent matrices are reused at every time step, which is why the unfolded network can be deep even when the cell definition is simple.
• The hidden state is the memory carrier. Each new state mixes the previous state and current input, so forgetting or preserving information becomes a structural issue.
• Vanishing gradient in RNNs has a temporal interpretation. The lecture is careful here: the whole gradient does not literally become zero. Rather, the terms corresponding to early inputs become negligible, so long-term dependencies are not learned.
• Gates create protected memory paths. LSTM and GRU keep part of the existing state and add new content on top of it, which creates shortcut-like paths for information and gradients.
• TBPTT is a computational compromise. It lowers memory use and often speeds training, but it may worsen the long-term memory problem because the backward horizon is shortened.
• Embeddings are trainable replacements for one-hot vectors. The lecture frames them as low-dimensional dense representations where geometric relations may become meaningful after training.

Mathematical Content

Base RNN equations.

For a BRNN with fully connected output, the lecture writes:

BPTT core equation.

Vanishing gradient mechanism. Because tanh'(u) ≤ 1, if the operator norm of the recurrent Jacobian stays below one, products of the form ∏ ∂hm/∂hm-1 shrink, so early-time contributions become negligible.

Exploding gradient countermeasure.

The lecture recommends gradient clipping and also mentions penalizations designed to preserve gradient norms over time.

LSTM.

The lecture's message is that the additive cell-state path mitigates structural vanishing/exploding effects, even if it does not magically remove all optimization difficulty.

GRU.

TBPTT.

The lecture highlights several configurations, including TBPTT(K, K) as full BPTT and TBPTT(k1, k1) as a common practical choice.

Embeddings.

Embeddings are dense, continuous, and trainable. Their axes have no fixed meaning a priori, but the geometry can become semantically useful after training.

Distinctions and Comparisons

• Deep feedforward vanishing gradient vs recurrent vanishing gradient. In RNNs, the lecture emphasizes loss of influence from early time steps rather than a blanket statement that "the gradient is zero".
• Standard RNN vs LSTM/GRU. Standard RNN fully refreshes hidden content each step; LSTM and GRU keep part of the previous content through gates and shortcut-style memory paths.
• BPTT vs TBPTT. Full BPTT is exact for the unfolded horizon but expensive; TBPTT is cheaper and lighter on memory but weaker on long dependencies.
• One-hot vs embedding. One-hot is sparse and high-dimensional with interpretable axes; embeddings are dense and low-dimensional with learned geometry.
• Autoencoder vs classifier. Autoencoders map data to itself for representation learning rather than directly predicting external labels.

Exam-Oriented Understanding

• For the hidden-state equation, write the recurrence explicitly and distinguish the hidden-state size from the number of hidden layers.
• For vanishing gradient in RNNs, explain the role of repeated Jacobian products and the resulting loss of long-term dependency learning.
• For TBPTT, mention both advantages and drawback: cheaper computation and memory, but more difficulty with long-term temporal dependencies.
• For LSTM/GRU, focus on the idea of gated memory preservation rather than trying to reproduce every gate diagram detail from memory.
• For embeddings, explain why one-hot vectors become impractical for large vocabularies and why dense trainable vectors help.

Source References

• Deep Learning Lecture 4 (2026_DL_4.pdf), slides 12-20: Base RNN, unfolded view, deep RNN, bi-directional RNN, recurrence equations.
• Deep Learning Lecture 4 (2026_DL_4.pdf), slides 21-45: BPTT equation, vanishing/exploding gradients, clipping, LSTM, GRU, comparison of gated models.
• Deep Learning Lecture 4 (2026_DL_4.pdf), slides 50-55: BPTT alternatives and truncated BPTT configurations.
• Deep Learning Lecture 4 (2026_DL_4.pdf), slides 57-73: 1-D convolutions, WaveNet, autoencoders, one-hot encoding, embeddings, neural-network pipeline advice.

Relevant Exam Alignment

The sample exam directly asks for the standard RNN hidden-state equation, the RNN-specific interpretation of vanishing gradients, and the definition plus tradeoff of truncated backpropagation through time.

The most important exam habit here is precision of wording: for RNNs, saying only "the gradient becomes small" is weaker than saying "terms linked to early time steps become negligible, so the network forgets long-range dependencies".