Deep Learning Training, Hyperparameters, and Regularization

Chapter Overview

This chapter moves from the bare existence of neural networks to the practical question of making them train and generalize well. The lecture revisits features and backpropagation, then studies design choices: number of layers, activation functions, output units, losses, initialization, validation, and explicit regularization methods.

It matters because many exam questions in the sample final are not about deriving a network from scratch but about choosing sensible ingredients, explaining why some pairings work and others fail, and naming ways to control overfitting.

The central theme is compatibility: task type, output layer, loss, activation, initialization, and regularization all have to fit together. A lot of weak exam answers come from naming a technique without tying it back to the objective it solves.

Important Definitions

• Hyperparameters. Architectural and optimization choices not learned directly by gradient descent, such as number of layers, number of neurons, learning-rate schedule, or regularization strength.
• Activation function. Nonlinearity applied after an affine transformation; examples in the lecture are step, sigmoid, tanh, ReLU, PReLU, ELU, maxout, and Swish.
• Dead neuron. A ReLU unit that outputs zero and stays inactive, making optimization harder.
• Output unit. The last-layer transformation matching the target type: linear, sigmoid, or softmax in the course material.
• Cross-entropy / negative log-likelihood. Classification loss comparing the predicted probability distribution to the empirical label distribution.
• Bias-variance decomposition. Error split into squared bias plus variance, with an extra noise term for prediction against noisy targets.
• Penalization. Add a complexity term to the training objective, typically only on weights in neural networks.
• Ridge / LASSO. ℓ2 and ℓ1 penalties, respectively.
• Dropout. Randomly remove units during training and rescale at prediction time.
• Batch normalization. Normalize activations within a batch, then learn affine parameters to restore flexible representations.
• Early stopping. Return the parameter values with lowest validation error instead of the latest iterate.

Key Concepts and Intuition

• Architecture choice is mostly empirical in the lecture. There is no universal rule for the number of layers or neurons; the recommended strategy is task-driven experimentation, pruning, and comparison.
• Activation tradeoffs are optimization tradeoffs. Step units are unusable because their gradient is zero almost everywhere; sigmoid and tanh saturate; ReLU is cheap and often faster but can die; newer variants modify the negative side.
• The output layer must encode the semantics of the target. Regression is not a classification problem with the wrong activation, and multiclass classification is not just many independent sigmoids in the lecture treatment.
• Initialization is about breaking symmetry without destroying optimization. Setting all weights equal makes neurons within a layer behave identically and wastes the architecture.
• Regularization is a family, not one technique. Penalization, validation-based model choice, dropout, batch normalization, and early stopping all target overfitting from different angles.

Mathematical Content

Activation functions emphasized in the lecture.

The lecture conclusion slide is pragmatic: prefer ReLU or Swish, test Leaky ReLU / maxout / ELU, possibly try tanh, and generally avoid sigmoid in hidden layers.

Output layers.

Losses.

The lecture explicitly warns that MSE should not be used with softmax output units.

Bias-variance decomposition.

Penalization.

Dropout algorithm.

For each mini-batch, sample Bernoulli masks for hidden and possibly input units, run forward and backward passes on the masked network, average the gradients, then update. At prediction time, use all units with weights multiplied by the probability of being kept.

Batch normalization.

Early stopping algorithm. Keep the parameter vector with lowest validation error, stop after a patience budget of repeated non-improvements, and return the best stored iterate rather than the last one.

Distinctions and Comparisons

• ReLU vs sigmoid/tanh. ReLU avoids positive-side saturation and is cheap, but can die on the negative side. Sigmoid and tanh saturate and can create vanishing gradients.
• Output unit vs loss. Regression naturally pairs with linear outputs and MSE/MAE; binary classification with sigmoid and cross-entropy; multiclass classification with softmax and cross-entropy.
• Ridge vs LASSO. Ridge shrinks continuously and reduces variance; LASSO can create sparsity but has no closed form in general.
• Dropout vs batch normalization. Dropout injects stochastic masking; batch normalization rescales activations and often reduces sensitivity to initialization and learning rate.
• Validation vs early stopping. Validation measures generalization for model choice; early stopping uses validation dynamics to stop training and keep the best iterate.

Exam-Oriented Understanding

• When asked about ReLU, mention both its benefit and its drawback. The lecture's exam-ready language is: cheap to compute, no saturation toward +infinity, but dead neurons can occur.
• For 0-1 loss, the key exam point is not "bad in practice" but "cannot be used to train with gradient-based methods because the derivative is zero almost everywhere".
• For dropout, distinguish training time from prediction time. This is one of the easiest places to lose precision.
• For regularization questions, do not list only penalization. The lectures explicitly present validation, penalization, dropout, batch normalization, and early stopping as distinct tools.
• For architecture-choice questions, say there is no universal formula and that the lecture recommends empirical task-driven tuning.

Source References

• Deep Learning Lecture 2 (2026_DL_2.pdf), slides 31-59: Hidden-layer size, activation functions, output units, classification losses, cross-entropy.
• Deep Learning Lecture 2 (2026_DL_2.pdf), slides 62-73: Weight initialization, overfitting, bias-variance decomposition.
• Deep Learning Lecture 2 (2026_DL_2.pdf), slides 77-90: Validation, penalization, ridge regression, LASSO, sparsity.
• Deep Learning Lecture 2 (2026_DL_2.pdf), slides 92-108: Dropout, batch normalization, early stopping, additional overfitting countermeasures.

Relevant Exam Alignment

The sample exam uses this chapter's material repeatedly: ReLU pros and cons, why 0-1 loss is unusable for training, detailed dropout procedure, four regularization techniques, and output-layer choices.

A strong answer links each design choice to its practical purpose: trainability, numerical stability, generalization, or task compatibility.