MAP545 Revision Site

Deep Learning Foundations

DL Foundations

← Optimization Foundations Back to Index Optimization Extensions I →

Chapter Overview

This chapter covers the deep-learning foundations introduced before the specialized architecture lectures: the supervised learning framework, loss and risk, Bayes predictors, validation, the historical perceptron, multilayer neural networks, logistic regression, multiclass outputs, and backpropagation.

It matters because the rest of the deep-learning material assumes you already know what a neuron computes, how a neural network is written layer by layer, and why backpropagation is the key computational ingredient rather than an optimizer by itself.

The course's path is deliberate: start from simple predictors, understand why the perceptron is historically important but limited, replace the hard threshold by smoother surrogates such as sigmoid and logistic loss, then scale to multilayer networks with backpropagation.

Important Definitions

  • Supervised learning setup. Data are pairs (X, Y) drawn i.i.d. from an unknown distribution D, with training set Dn = {(X1, Y1), ..., (Xn, Yn)}.
  • Predictor. A function f : X -> Y chosen from a class F.
  • Loss. ℓ(Y, f(X)) measures prediction quality for one example.
  • Risk. R(f) = E[ℓ(Y, f(X))], the expected loss on a fresh sample from the data distribution.
  • Bayes predictor. The predictor f* minimizing the conditional expected loss for each input x.
  • Neuron. A unit computing a weighted sum of inputs, adding a bias, then applying an activation function.
  • Perceptron. A binary classifier using the step activation 1z > 0 on an affine score.
  • Logistic regression. Binary classification model using a sigmoid output and logistic/cross-entropy loss.
  • Backpropagation. Efficient recursive computation of gradients in a layered network by reusing intermediate chain-rule terms.
  • Multiclass logistic regression. Softmax-based generalization assigning one score vector and one probability distribution over classes.

Key Concepts and Intuition

  • Learning is defined through loss, not through intuition alone. The lectures insist that "good predictor" only becomes meaningful after choosing a loss and hence a risk.
  • The Bayes predictor is a reference point, not a trainable object. It is the theoretical best rule for the chosen loss, but it depends on the unknown data distribution.
  • The perceptron is historically foundational but practically too rigid. It assumes a linear decision boundary and separable data, which the lecture explicitly treats as too restrictive.
  • Going deep means composing affine maps and nonlinearities. Once one hidden layer is allowed, the network can represent functions a single perceptron cannot, such as XOR-style logic.
  • Backpropagation is an efficiency breakthrough. The point is not just that chain rule works, but that it can be organized so the whole gradient is computed in one forward/backward sweep instead of one pass per parameter.

Mathematical Content

Learning framework.

\[R(f) = \mathbb{E}[ \ell(Y, f(X)) ]\]
\[0-1 loss: \ell(Y, f(X)) = 1_{Y \neq f(X)}, quadratic loss: \ell(Y, f(X)) = |Y - f(X)|^{2}\]
\[\text{Bayes predictor:} f^*(x) \in \operatorname{argmin}_{p \in Y} \mathbb{E}[ \ell(Y, p) \mid X = x ]\]
\[\text{For binary 0-1 classification:} f^*(X) = +1 if P(Y=+1 \mid X) \ge 1/2, otherwise -1\]
\[\text{For quadratic regression:} f^*(X) = \mathbb{E}[Y \mid X]\]

Neuron and perceptron.

\[a = \sigma( \sum_{j=1}^{d} w_{j}x_{j} + b )\]
\[\sigma(z) = 1_{z > 0}\]
\[\text{Objective in perceptron notation:} y_{i} \langle w~, x~_{i} \rangle \ge 0\]
\[\text{Perceptron update:} if y_{i} \langle w~, x~_{i} \rangle \le 0, then w~ \leftarrow w~ + y_{i}x~_{i}\]
\[\text{Perceptron surrogate loss:} \ell_{i}(w~) = \max(0, -y_{i}\langle w~, x~_{i} \rangle)\]

The lecture derives a finite-update bound in the separable case: if the margin is γ and the inputs are bounded by R, the number of effective updates is bounded by (1 + R2) / γ2.

Logistic regression.

\[\sigma(z) = 1 / (1 + e^{-z})\]
\[\text{logistic loss:} \ell(y, y^) = \log(1 + \exp(-y y^))\]
\[\min_{w,b} (1/n) \sum_{i=1}^{n} \log(1 + \exp(-y_{i}(\langle w, x_{i} \rangle + b)))\]
\[P(Y = 1 \mid X = x) = 1 / (1 + \exp(-\langle w, x \rangle - b))\]

Layered neural network notation.

\[z^{(k)} = W^{(k)}a^{(k-1)} + b^{(k)}, a^{(k)} = \sigma( z^{(k)} )\]

Backpropagation equations.

\[\delta^{(L)} = \sigma'(z^{(L)}) \odot (\partial\ell/\partial a)(y, a^{(L)})\]
\[\delta^{(k)} = (W^{(k+1)})^{T}\delta^{(k+1)} \odot \sigma'(z^{(k)})\]
\[\partial C / \partial b^{(k)}_{j} = \delta^{(k)}_{j}, \partial C / \partial w^{(k)}_{i,j} = a^{(k-1)}_{j} \delta^{(k)}_{i}\]

Training loop. Initialize weights and biases randomly, run feedforward on the batch, compute the batch loss, run backpropagation from the last layer to the first, then update parameters with a gradient-based optimizer.

Multiclass outputs.

\[P(Y = y \mid X = x) = \exp(\langle w~_{y}, x \rangle) / \sum_{k=1}^{K} \exp(\langle w~_{k}, x \rangle)\]
\[- log-likelihood = - \sum_{i=1}^{n} \log P(Y = y_{i} \mid X = x_{i})\]

Distinctions and Comparisons

  • Loss vs risk. Loss is per-sample; risk is the average over the unknown data distribution.
  • Perceptron vs logistic regression. The perceptron uses a hard threshold and linear separability assumptions; logistic regression uses a probabilistic sigmoid model and a smooth optimization objective.
  • Backpropagation vs optimization. Backpropagation computes gradients; GD, SGD, Adam, and so on use those gradients to update parameters.
  • Binary sigmoid vs multiclass softmax. A binary task maps naturally to one sigmoid probability; a multiclass task needs a normalized vector over classes.
  • Validation vs test. Validation is for model choice; test is the final assessment and must not be used as a tuning signal.

Exam-Oriented Understanding

  • For a neuron, always say all three components: weighted sum, bias, activation.
  • For the perceptron algorithm, give the update equation and say when it is triggered. Also say clearly that the algorithm is limited to a single linear threshold unit and separable data.
  • For backpropagation, mention the naive alternative: computing each gradient coordinate separately would require a number of passes proportional to the number of parameters.
  • For output-layer questions, tie the answer to the task: linear for regression, sigmoid for binary classification, softmax for multiclass classification.
  • A common incomplete answer is to write formulas without explaining what quantity is being predicted or minimized.

Source References

  • Deep Learning Lecture 1 (2026_DL_1.pdf), slides 8-13: Supervised learning framework, losses, risk, Bayes predictor, validation.
  • Deep Learning Lecture 1 (2026_DL_1.pdf), slides 29-42: Neuron, perceptron, perceptron algorithm, convergence, limits.
  • Deep Learning Lecture 1 (2026_DL_1.pdf), slides 56-63: One hidden layer, logistic regression, relation to smooth classification losses.
  • Deep Learning Lecture 1 (2026_DL_1.pdf), slides 71-81: Backpropagation equations and feedforward/backprop training loop.
  • Deep Learning Lecture 1 (2026_DL_1.pdf), slides 84-86: Multiclass logistic regression via maximum likelihood.
Only lecture PDFs are used as content sources here. The sample exam is used only to choose emphasis and exam-answer style.

Relevant Exam Alignment

The sample exam heavily targets this chapter: neuron operation, perceptron update, input/output layer choice, why backpropagation improves on a naive gradient computation, why perceptron does not train deep networks, and the exact role of the activation function.

The safest exam strategy is to answer in a sentence-first format: define the object, write the key equation, then state the limitation or interpretation.

← Optimization Foundations Back to Index Optimization Extensions I →