By the end of this lecture, you should be able to:
This final module will focus on the most popular class of models in machine learning today: neural networks, deep learning, and generative AI. Briefly, we will cover:
Neural networks were originally inspired by biological neural networks in the brain.
Artificial neural networks are machine learning models that are loosely inspired by this biological idea.
And that’s where the neuroscience connection ends. From now on, we’re back to math and statistics!
In the last lecture, we discussed boosted ensembles, which composed simple “weak learners” \(h_1, \ldots, h_j\) (e.g. bump functions or small decision trees) into ensembles of the form:
\[ \hat{f}_\mathcal{D}(X) = \sum_{j=1}^M \beta_j \hat h_j(X), \]
Each weak learner \(h_j\) has a set of learnable parameters \(\theta_j\) (e.g. split points in trees, location of the “bump”), so we can also write the ensemble like:
\[ \hat{f}_\mathcal{D}(X) = \sum_{j=1}^M \beta_j \hat h(X; \theta_j), \]
where \(\hat h(X; \theta_j)\) is the \(j^\mathrm{th}\) weak learner with parameters \(\theta_j\). Crucially, \(\hat h(X; \theta_j)\) applies some simple non-linear transformation to the input \(X\) to weakly predict the residual response, and the ensemble sums up these non-linear transformations to produce a final prediction.
Earlier in the course, we discussed a similar class of predictive models.
Fixed basis expansions!
Let \(\hat h(X; \theta_j)\) represent a polynomial basis function, where \(\theta_j\) represents the degree of the polynomial, i.e.
\[ \hat h(X; \theta_j) = X^{\theta_j}. \]
We can also think of \(\hat h(X; \theta_j)\) as a “weak learner” of sorts, and we combine \(m\) such weak learners with different degrees to form a polynomial regression model:
\[ \hat{f}_\mathcal{D}(X) = \sum_{j=1}^M \beta_j X^{\theta_j} = \sum_{j=1}^M \beta_j \hat h(X; \theta_j). \]
Both models learn the \(\beta_j\) weights from data. However, there are two key differences between boosted ensembles and fixed basis expansions:
In boosted ensembles, the parameters of the weak learners \(\theta_j\) are learned from data during training. In fixed basis expansions, the parameters \(\theta_j\) are fixed ahead of time (e.g. polynomial degrees); only the \(\beta_j\) weights are learned from data.
In fixed basis expansions, the \(\beta_j\) weights are learned simultaneously via a single optimization problem (e.g. least squares). In boosted ensembles, the \(\beta_j\) weights and weak learner parameters \(\theta_j\) are learned sequentially via an iterative procedure.
What challenges might arise when learning all parameters simultaneously?
Both of these potential concerns prevented statisticians from taking neural networks seriously for many years. However, in practice, neither of these issues are as problematic as one might expect. Us statisticians are only starting to understand why this is the case!
Neural networks have the same functional form as boosted ensembles/basis regressions:
\[ \hat{f}_\mathcal{D}(X) = \sum_{j=1}^M \beta_j \hat h(X; \theta_j), \]
where the weak learner/learned basis function \(\hat h(X; \theta_j)\) takes a specific parametric form:
\[ \hat h(X; \theta_j) = g\left( \theta_{j0} + X_1 \theta_{j1} + \ldots + X_{p} \theta_{jp} \right) = g\left( X^\top \theta \right)\]
:::{.callout-note collapse=“true” title=“Implicit bias term”} As we have throughout this course, we often drop the intercept term \(\theta_{j0}\) for notational convenience, and we assume that \(X\) has been augmented with a column of ones to account for the intercept. :::
\(\hat h(X; \theta_j)\) is called a neuron or hidden unit
\(\theta_j \in \mathbb{R}^p\) are the learned parameters, or weights of the neuron, and
\(g: \mathbb{R} \to \mathbb{R}\) is a fixed non-linear activation function, typically chosen to be the Rectified Linear Unit (ReLU) function:
\[ g(z) = \max(0, z). \]
\(\phi(X) = \begin{bmatrix} g(X^\top \theta_1) \\ g(X^\top \theta_2) \\ \vdots \\ g(X^\top \theta_M) \end{bmatrix}\) is called the hidden feature representation, or the activations of \(X\).
The final prediction is a linear combination of these activations:
\[ \hat{f}_\mathcal{D}(X) = \phi(X)^\top \beta, \quad \beta = \begin{bmatrix} \beta_1 \\ \vdots \\ \beta_M \end{bmatrix}. \]
This functional form: a linear combination of non-linear transformations of the input, looks just like basis regression. Again, the key difference is that the basis features \(\phi(X)\) are learned from data, rather than using a fixed basis expansion.
The activation function \(g(z) = \max(0, z)\) introduces non-linearity into the model. It produces a non-linear basis feature of \(X\).
Consider the case with one-dimensional data:
\[ g(\theta_{j0} + X \theta_{j1}) = \max(0, \theta_{j0} + X \theta_{j1}). \]
library(ggplot2)
relu <- function(x, beta0, beta1) pmax(0, beta0 + beta1 * x)
x_vals <- seq(-3, 3, length.out = 100)
y_vals <- relu(x_vals, beta0 = 1, beta1 = 2)
ggplot(data.frame(x = x_vals, y = y_vals), aes(x, y = y)) +
geom_line(size = 1.5, color = "blue") +
labs(title = "ReLU Activation Function (β0 = 1, β1 = 2)", x = "Input", y = "Output") +
theme_minimal()
This is similar to a linear spline basis function with a knot at \(x = -\theta_{j0} / \theta_{j1}\).
By forming a linear combination of many such ReLU basis functions, each with different offset and slope parameters \(\theta_j\), we end up with a piecewise linear function:
library(tidyverse)
library(ggplot2)
relu <- function(x, beta0, beta1) pmax(0, beta0 + beta1 * x)
x_vals <- seq(-3, 3, length.out = 100)
unit1 <- relu(x_vals, beta0 = 1, beta1 = 1)
unit2 <- relu(x_vals, beta0 = -1, beta1 = 1)
unit3 <- relu(x_vals, beta0 = 0.5, beta1 = 1)
combined <- 0.5 * unit1 + (-1) * unit2 + 1.5 * unit3
df_units <- tibble(x = x_vals, unit1 = unit1, unit2 = unit2, unit3 = unit3) |>
pivot_longer(-x, names_to = "unit", values_to = "y")
p1 <- ggplot(df_units, aes(x, y, color = unit)) +
geom_line(size = 1.2) +
labs(title = "Individual ReLU Units", x = "Input", y = "Output") +
theme_minimal()
p2 <- ggplot(data.frame(x = x_vals, y = combined), aes(x, y = y)) +
geom_line(size = 1.5, color = "blue") +
labs(title = "Combined NN Output", x = "Input", y = "Output") +
theme_minimal()
library(gridExtra)
grid.arrange(p1, p2, ncol = 2)
What if we used the identity activation function \(g(z) = z\) instead of ReLU?
The model would reduce to a standard linear regression model:
\[ \hat{f}_\mathcal{D}(X) = \sum_{j=1}^M \beta_j (X^\top \theta_j) = X^\top \left( \sum_{j=1}^M \beta_j \theta_j \right). \]
The model would lose its ability to capture non-linear relationships in the data.
While we have been using mathematical notation to describe neural networks, you will often see them represented by a toy diagram:
Each circle represents a neuron
Each solid arrow represents a learned parameter
Each dotted arrow represents an application of the activation function \(g(\cdot)\)
\(\Theta \in \mathbb R^{m \times p}\) is the concatenation of all the \(\theta_j \in \mathbb R^p\) weights
\(A_1, \ldots, A_m\) are the activations of each neuron, so that \(\phi(X) = \begin{bmatrix} A_1 \\ \vdots \\ A_m \end{bmatrix}\).
\(P_1, \ldots, P_m\) are often referred to as the pre-activations, i.e. the linear combination before applying the activation function:
\[ P_j = X^\top \theta_j, \quad A_j = g(P_j). \]
This diagram shows how the input covariates are “processed.” It will become especially useful when we discuss deeper neural networks with multiple hidden layers as well as the training procedure for neural networks.
The key idea is to recursively construct basis functions of the form:
\[ \phi^{(1)}(X) = \begin{bmatrix} g\left( X^\top \theta_1^{(1)} \right) \\ g\left( X^\top \theta_2^{(1)} \right) \\ \vdots \\ g\left( X^\top \theta_{M_1}^{(1)} \right) \end{bmatrix}, \quad \phi^{(L+1)}(X) = \begin{bmatrix} g\left( \phi^{(L)}(X)^\top \theta_1^{(L+1)} \right) \\ g\left( \phi^{(L)}(X)^\top \theta_2^{(L+1)} \right) \\ \vdots \\ g\left( \phi^{(L)}(X)^\top \theta_{M_{L+1}}^{(L+1)} \right) \end{bmatrix} \]
This idea is perhaps best understood through the diagram representation:
Here, we have multiple hidden layers \(\ell \in 1, \ldots, L\), each with its own set of:
The final prediction is still a linear combination of the last layer’s activations, i.e. it is a basis regression model with the basis features \(\phi^{(L)}(X)\):
\[ \hat f_{\mathcal D}(X) = \phi^{(L)}(X)^\top \beta. \]
However, the basis features \(\phi^{(L)}(X)\) are recursively constructed through a non-linear combination of previous basis function \(\phi^{(L-1)}(X)\)
The key insight: depth allows us to compose non-linear transformations, leading to exponentially more expressive functions.
Consider a simple example with one-dimensional input:
Now consider what happens when we add a second hidden layer:
More precisely:
Exponential growth! With the same total number of parameters, a deep network can represent exponentially more complex functions than a shallow one.
This hierarchical processing mirrors how our brains work, and it’s why depth is so powerful for complex tasks like vision and language understanding.
Consider a neural network with:
With these parameter counts in mind, we can see that depth is much more efficient than width for increasing expressivity.