Overview
Neural networks are models for supervised learning
Linear combinations of features are passed through a non-linear transformation in successive layers
At the top layer, the resulting latent factors are fed into an algorithm for predictions
(Most commonly via least squares or logistic loss)
Background
Neural networks have come about in 3 “waves”
The first was an attempt in the 1950s to model the mechanics of the human brain
It appeared the brain worked by
- taking atomic units known as neurons, which can be “on” or “off”
- putting them in networks
A neuron itself interprets the status of other neurons
There weren’t really computers, so we couldn’t estimate these things
Background
After the development of parallel, distributed computation in the 1980s, this “artificial intelligence” view was diminished
And neural networks gained popularity
But, the growing popularity of SVMs and boosting/bagging in the late 1990s, neural networks again fell out of favor
This was due to many of the problems we’ll discuss (non-convexity being the main one)
State-of-the-art performance on various classification tasks has been accomplished via neural networks
Today, Neural Networks/Deep Learning are the hottest…
Recall nonparametric regression
Suppose \(Y \in \mathbb{R}\) and we are trying estimate the regression function \[\Expect{Y\given X} = f_*(X)\]
In Module 2, we discussed basis expansion,
We know \(f_*(x) =\sum_{k=1}^\infty \beta_k \phi_k(x)\) some basis \(\phi_1,\phi_2,\ldots\)
Truncate this expansion at \(K\): \(f_*^K(x) \approx \sum_{k=1}^K \beta_k \phi_k(x)\)
Estimate \(\beta_k\) with least squares
Recall nonparametric regression
The weaknesses of this approach are:
- The basis is fixed and independent of the data
- If \(p\) is large, then nonparametrics doesn’t work well at all (recall the Curse of Dimensionality)
- If the basis doesn’t “agree” with \(f_*\), then \(K\) will have to be large to capture the structure
- What if parts of \(f_*\) have substantially different structure? Say \(f_*(x)\) really wiggly for \(x \in [-1,3]\) but smooth elsewhere
An alternative would be to have the data tell us what kind of basis to use (Module 5)
1-layer for Regression
A single layer neural network model is \[
\begin{aligned}
&f(x) = \sum_{k=1}^K \beta_k h_k(x) \\
&= \sum_{k=1}^K \beta_k \ g(w_k^{\top}x)\\
&= \sum_{k=1}^K \beta_k \ A_k\\
\end{aligned}
\]
Compare: A nonparametric regression \[f(x) = \sum_{k=1}^K \beta_k {\phi_k(x)}\]
Terminology
\[f(x) = \sum_{k=1}^{{K}} {\beta_k} {g( w_k^{\top}x)}\] The main components are
- The derived features \({A_k = g(w_k^{\top}x)}\) and are called the hidden units or activations
- The function \(g\) is called the activation function (more on this later)
- The parameters \({\beta_k},{w_k}\) are estimated from the data for all \(k = 1,\ldots, K\).
- The number of hidden units \({K}\) is a tuning parameter
\[f(x) = \sum_{k=1}^{{K}} \beta_0 + {\beta_k} {g(w_{k0} + w_k^{\top}x)}\]
- Could add \(\beta_0\) and \(w_{k0}\). Called biases (I’m going to ignore them. It’s just an intercept)
Terminology
\[f(x) = \sum_{k=1}^{{K}} {\beta_k} {g(w_k^{\top}x)}\]
Notes (no biases):
\(\beta \in \R^k\)
\(w_k \in \R^p,\ k = 1,\ldots,K\)
\(\mathbf{W} \in \R^{K\times p}\)
What about classification (10 classes, 2 layers)
\[
\begin{aligned}
A_k^{(1)} &= g\left(\sum_{j=1}^p w^{(1)}_{k,j} x_j\right)\\
A_\ell^{(2)} &= g\left(\sum_{k=1}^{K_1} w^{(2)}_{\ell,k} A_k^{(1)} \right)\\
z_m &= \sum_{\ell=1}^{K_2} \beta_{m,\ell} A_\ell^{(2)}\\
f_m(x) &= \frac{1}{1 + \exp(-z_m)}\\
\end{aligned}
\]
Predict class with largest probability \(\longrightarrow\ \widehat{Y} = \argmax_{m} f_m(x)\)
What about classification (10 classes, 2 layers)
Notes:
\(B \in \R^{M\times K_2}\) (here \(M=10\)).
\(\mathbf{W}_2 \in \R^{K_2\times K_1}\)
\(\mathbf{W}_1 \in \R^{K_1\times p}\)
Two observations
- The \(g\) function generates a feature map
We start with \(p\) covariates and we generate \(K\) features (1-layer)
Logistic / Least-squares with a polynomial transformation
\[
\begin{aligned}
&\Phi(x) \\
& =
(1, x_1, \ldots, x_p, x_1^2,\ldots,x_p^2,\ldots\\
& \quad \ldots x_1x_2, \ldots, x_{p-1}x_p) \\
& =
(\phi_1(x),\ldots,\phi_{K_2}(x))\\
f(x) &= \sum_{k=1}^{K_2} \beta_k \phi_k(x) = \beta^\top \Phi(x)
\end{aligned}
\]
Neural network
\[\begin{aligned}
A_k &= g\left( \sum_{j=1}^p w_{kj}x_j\right) = g\left( w_{k}^{\top}x\right)\\
\Phi(x) &= (A_1,\ldots, A_K)^\top \in \mathbb{R}^{K}\\
f(x) &=\beta^{\top} \Phi(x)=\beta^\top A\\
&= \sum_{k=1}^K \beta_k g\left( \sum_{j=1}^p w_{kj}x_j\right)\end{aligned}\]
Two observations
- If \(g(u) = u\), (or \(=3u\)) then neural networks reduce to (massively underdetermined) ordinary least squares (try to show this)
- ReLU is the current fashion (used to be tanh or logistic)
![]()
Next time…
How do we estimate these monsters?
