Recall: Bias/Variance Tradeoff For Trees
Neural networks have lots of parameters ( \(O(D^2 L)\), which is typically \(> n\) )
In theory, we would expect similar bias/variance curves for neural networks as a function of
# params
The Surprising Bias/Var Curves For NN (Double Descent)
NN risk (as a function of
# params) experiences a “double descent” shape?!?!?!Most modern NN have tons of parameters, and so they’re explained by the right side of the graph
The Surprising Bias/Var Curves For NN (Double Descent)
Image credit: Belkin et al., (2019)
Double descent is a newly discovered phenomenon (~2019)
Statisticians are still trying to understand why it occurs.
There has been good progress since ~2020!
To Understand Double Descent: Study Basis Regression
The double descent phenomenon is not specific to neural networks.
We can observe it in basis regression (read: linear models!) as we increase the number of basis functions \(> n\):
xn <- seq(-1.5 * pi, 1.5 * pi, length.out = 1000)
# Spline by hand
X <- bs(df$x, df = 20, intercept = TRUE)
Xn <- bs(xn, df = 20, intercept = TRUE)
S <- svd(X)
yhat <- Xn %*% S$v %*% diag(1/S$d) %*% crossprod(S$u, df$y)
g + geom_line(data = tibble(x = xn, y = yhat), colour = orange) +
ggtitle("20 basis functions (n=20)")
xn <- seq(-1.5 * pi, 1.5 * pi, length.out = 1000)
# Spline by hand
X <- bs(df$x, df = 40, intercept = TRUE)
Xn <- bs(xn, df = 40, intercept = TRUE)
S <- svd(X)
yhat <- Xn %*% S$v %*% diag(1/S$d) %*% crossprod(S$u, df$y)
g + geom_line(data = tibble(x = xn, y = yhat), colour = orange) +
ggtitle("40 basis functions (n=20)")
Code
doffs <- 4:50
mse <- function(x, y) mean((x - y)^2)
get_errs <- function(doff) {
X <- bs(df$x, df = doff, intercept = TRUE)
Xn <- bs(xn, df = doff, intercept = TRUE)
S <- svd(X)
yh <- S$u %*% crossprod(S$u, df$y)
bhat <- S$v %*% diag(1 / S$d) %*% crossprod(S$u, df$y)
yhat <- Xn %*% S$v %*% diag(1 / S$d) %*% crossprod(S$u, df$y)
nb <- sqrt(sum(bhat^2))
tibble(train = mse(df$y, yh), test = mse(yhat, sin(xn)), norm = nb)
}
errs <- map(doffs, get_errs) |>
list_rbind() |>
mutate(`degrees of freedom` = doffs) |>
pivot_longer(train:test, values_to = "error")
ggplot(errs, aes(`degrees of freedom`, error, color = name)) +
geom_line(linewidth = 2) +
coord_cartesian(ylim = c(0, .12)) +
scale_x_log10() +
scale_colour_manual(values = c(blue, orange), name = "") +
geom_vline(xintercept = 20)
Inflection point occurs when
# basis functions = n!This is the point at which our basis regressor is able to perfectly fit the training data.
Understanding Double Descent (Less Hand-Wavy)
(From Hastie et al., 2020)
\(\gamma = D / N\) (ratio of features / data)
\(\sigma^2 = \mathbb{E}[Y|X]\) (observational noise)
When basis features are uncorrelated, we have (asymptotically)
\[ \begin{aligned} \mathrm{Bias}^2 &= \begin{cases} 0 & \gamma < 1 \text{ (underparam.)} \\ 1 - \tfrac{1}{\gamma} & \gamma \geq 1 \text{ (overparam.)} \end{cases} \\ & \\ \mathrm{Var} &= \begin{cases} \sigma^2 \tfrac{\gamma}{1 - \gamma} & \gamma < 1 \text{ (underparam.)} \\ \sigma^2 \tfrac{1}{\gamma - 1} & \gamma \geq 1 \text{ (overparam.)} \end{cases} \\ \end{aligned} \]
Specialty architectures: convolutional neural networks
Rather than computing an inner product with the hidden layer parameters (i.e. \(\boldsymbol w^{(i)\top}_j \boldsymbol a^{(t - 1)}\)), we instead perform a convolution:
\[ \boldsymbol a^{(t)}_j = g( \boldsymbol w^{(i)}_j \star \boldsymbol a^{(t - 1)}) \]
Captures spatial correlations amongst neighbouring pixels
Predictions remain constant even if we translate objects in the image
The convolutional building blocks are usually combined with other building blocks, like pooling layers and normalization layers.
Specialty architectures: convolutional neural networks
Why is a convolutional neural network better for images?
Image credit: Varsha Kishore
With an standard MLP, we’d need to “flatten” our image into a vector of pixels. This flattening doesn’t preseve spatial correlations amongst pixels.
If the dog in our image shifts, then we are not guaranteed to make the same prediction (we are not translation invariant).
Transfer Learning
You want to build an image classifier for CT scans, but you only have \(n=1000\) 😢
Conventional wisdom would tell you that you don’t have enough data to train a neural network.
Transfer learning to the rescue!
Start with an existing neural network trained on a related predictive task
Train this neural network on your data using gradient descent with a small step size
Also known as fine-tuning
Transfer Learning
Why Does This Work?
The original NN has learned basis functions that are REALLY good for image data
You are now essentially using these good basis functions on your smaller dataset