MAP545 Revision Site

MAP545 Revision Synthesis

A navigable revision website built only from the local lecture PDFs, with exam emphasis guided by the sample final.

Purpose

This mini-site is an exam-oriented revision synthesis for MAP545, built only from the seven lecture PDFs in the folder for course content. It is deliberately longer and more explicit than the slides: definitions are unpacked, formulas are named, algorithms are contextualized, and the local exam PDFs are used to drive the recap and the practice tooling.

How This Synthesis Is Organized

The navigation follows lecture chronology, not topic clustering by discipline.
Each chapter page follows the same template: overview, definitions, intuition, mathematical content, comparisons, exam-oriented understanding, source references, and exam alignment.
When optimization material overlaps across lectures, the fullest explanation is kept at the earliest natural chapter and later chapters focus on extensions or wrap-up insights.
The exam recap page uses the local blank exam and corrected exam as exam-specific sources, but it explicitly flags any corrected item that is not clearly supported by the lecture PDFs.
The quiz trainer is lecture-backed and exam-shaped: the lectures provide the content, while the exam PDFs guide emphasis and answer style.

Course Structure Summary

Optimization Track

Foundations and geometry of optimization, GD and SGD, then extensions: acceleration, variance reduction, Newton-type methods, coordinate methods, adaptivity, and practical deep-learning optimizers.

Deep Learning Track

Supervised learning and neural-network basics, training ingredients and regularization, then specialized architectures: CNNs for images and RNNs/embeddings for sequences.

Shared Thread

Backpropagation supplies gradients; optimization determines how these gradients are used. The course repeatedly links architecture design to optimizer behavior, conditioning, and generalization.

Chapter 01

Optimization Foundations and Basic Descent Methods

Optimization Foundations. Chronological position in the course: 1 / 7.

Chapter 02

Deep Learning Foundations

DL Foundations. Chronological position in the course: 2 / 7.

Chapter 03

Optimization Extensions I: Acceleration, Variance Reduction, and Newton-Type Methods

Optimization Extensions I. Chronological position in the course: 3 / 7.

Chapter 04

Deep Learning Training, Hyperparameters, and Regularization

DL Training and Regularization. Chronological position in the course: 4 / 7.

Chapter 05

Convolutional Neural Networks

CNNs. Chronological position in the course: 5 / 7.

Chapter 06

Recurrent Neural Networks, Sequence Training, and Embeddings

RNNs and Embeddings. Chronological position in the course: 6 / 7.

Chapter 07

Optimization Extensions II and Deep-Learning Optimizers

Optimization Extensions II. Chronological position in the course: 7 / 7.

Exam Tools

Exam Tool

Exam Recap

Question-by-question corrected exam walkthrough with lecture links, scope flags, and cleaned explanations for every item.

Practice Tool

Quiz Trainer

Interactive lecture-backed practice bank with open-response prompts, auto-checked objective items, and local browser persistence.

Source Policy

Course content source PDFs: APM52445_Optimization_2026_Lect1_annot.pdf, APM52445-Optimization-Lect2.pdf, APM52445-Optimization-Lect3.pdf, 2026_DL_1.pdf, 2026_DL_2.pdf, 2026_DL_3.pdf, and 2026_DL_4.pdf.

Exam PDFs: Exam_Map545.pdf for original wording and Final Optimization.pdf for correction guidance. These are used on the exam recap page and to shape the quiz trainer, not to silently rewrite lecture-backed course content.

EXAM MUST KNOW

1. Definitions to know by heart

Risk, loss, Bayes predictor, validation/test split, empirical risk minimization.
Neuron, perceptron, logistic regression, softmax, backpropagation.
Convexity, L-smoothness, μ-strong convexity, condition number.
Convolution, pooling, weight sharing, feature map, flatten layer.
RNN hidden state, BPTT, TBPTT, LSTM, GRU, embedding.

2. Key formulas / update rules / identities to know

\[\text{Perceptron update:} w~ \leftarrow w~ + y_{i}x~_{i} when y_{i}\langle w~, x~_{i} \rangle \le 0\]

\[\text{GD:} \theta_{t+1} = \theta_{t} - \eta\nabla f(\theta_{t})\]

\[\text{SGD:} \theta_{t} = \theta_{t-1} - \eta_{t}g_{t}(\theta_{t-1})\]

\[\text{Backprop:} \delta^{(k)} = (W^{(k+1)})^{T}\delta^{(k+1)} \odot \sigma'(z^{(k)})\]

\[\text{Softmax:} P(Y=i \mid X=x) = \exp(z_{i}) / \sum_{k} \exp(z_{k})\]

\[\text{Newton:} \theta_{t+1} = \theta_{t} - (\nabla^{2}f(\theta_{t}))^{-1}\nabla f(\theta_{t})\]

\[\text{Heavy Ball:} \theta_{t+1} = \theta_{t} - \eta_{t}\nabla f(\theta_{t}) + \beta_{t}(\theta_{t} - \theta_{t-1})\]

\[\text{RNN:} h_{t} = \tanh(W_{HH}h_{t-1} + W_{IH}x_{t} + b_{h})\]

\[\text{LSTM memory path:} c_{t} = f_{t} \odot c_{t-1} + i_{t} \odot g_{t}\]

3. Core concepts most likely to be tested

Why backpropagation is computationally efficient and why it is not an optimization algorithm.
Why CNNs fit images better than dense MLPs.
Why ReLU is usually preferred to sigmoid/tanh in hidden layers, and what dead neurons are.
What conditioning means geometrically and why momentum / Newton-type methods help.
Why SGD is favored in ML despite weaker asymptotic accuracy than GD.
Why gated RNNs help with long-range dependencies.

4. Typical exam reasoning patterns

Define the object, then give the equation, then give the interpretation.
Compare two nearby concepts by naming the exact difference in trainable parameters, computational cost, or gradient behavior.
State assumptions before stating convergence rates.
For architecture questions, tie the choice to the data structure: image, sequence, regression, or classification.
For optimizer questions, mention both the advantage and the drawback.

5. Common traps and confusions

Backpropagation is not GD, SGD, Adam, or any optimizer. It is a gradient-computation procedure.
Perceptron algorithm is not a training method for deep networks.
0-1 loss is not a practical gradient-based training loss.
Pooling and flatten layers do not introduce trainable parameters.
In RNNs, vanishing gradient refers especially to early-time contributions becoming negligible.
Output-layer choice depends on the prediction target; there is no one-size-fits-all last layer.

6. Minimum knowledge to pass

Chapter 1. Definitions of convexity/smoothness/strong convexity, GD and SGD updates, and the practical GD versus SGD tradeoff.
Chapter 2. Supervised learning setup, neuron operation, perceptron update, logistic regression idea, and what backpropagation does.
Chapter 3. Heavy Ball versus NAG, basic idea of variance reduction, and Newton update.
Chapter 4. ReLU versus sigmoid/tanh, task-compatible output units, at least four regularization techniques, and dropout train/test distinction.
Chapter 5. Convolution versus pooling, why CNNs suit images, and the main signatures of AlexNet / GoogLeNet / ResNet.
Chapter 6. Standard RNN recurrence, vanishing gradient in the recurrent setting, and TBPTT.
Chapter 7. Coordinate/adaptive method intuition and the difference between Adagrad, RMSprop, and Adam.

7. Recurring themes suggested by the example exam

The exam rewards phrase-based explanations, not bare formulas.
Definitions, update equations, and "why this method / layer / activation?" questions are central.
Comparison questions are common: GD versus SGD, convolution versus pooling, RNN versus LSTM/GRU, SAG versus SVRG.
Common zero-point mistakes are explicitly hinted by the exam: confusing backpropagation with optimization, ignoring train/test differences in dropout, or omitting assumptions behind rates.