Lecture 2: Introduction to Learning, Regression
Learning Objectives
By the end of this lecture, you will be able to:
- Distinguish between “learning,” “supervised learning,” “prediction,” and “inference.”
- Translate between algorithmic and statistical perspectives on learning
- Define a statistical model
- Define a hypothesis class
- Define an estimator
- Define a prediction rule
- Define the standard linear regression model
- Derive ordinary least squares from the empirical risk minimization (ERM) framework (and, optionally, connect it to maximum likelihood estimation)
What is (Supervised) Learning?
There are many formulations of “learning” in statistics and machine learning. The main focus of this course is supervised learning.
- Goal of supervised learning: predict a response variable \(Y\) given a set of covariates \(X\)
- For example, predicting house prices based on features like size, location, and number of bedrooms.
Learning from data:
We are given a training dataset: \[ \mathcal{D} = \{(X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)\} \in (\mathbb R^p \times \mathcal Y)^n \]
where:
- \(X_i \in \mathbb R^p\) are the \(p\)-dimensional covariates (e.g. size, location, number of bedrooms, etc. for each house)
- \(Y_i \in \mathcal Y\) is the response variable (e.g. house price)
- \(\mathcal Y\) is the space of possible responses.
- For our problem and other regression problems, \(\mathcal Y = \mathbb R\).
- For classification problems (i.e. predicting whether the house is for sale or not) \(\mathcal Y\) is a finite set of classes (e.g. “for sale” vs “not for sale”).
From this training set, we want to learn a function \(\hat f : \mathbb R^p \to \mathcal Y\) that accurately predicts the response for new observations.
- Given a set of new covariates \(X_\mathrm{new}\) (e.g. a new house that we don’t know the price of)
- We predict \(\hat Y_\mathrm{new} = \hat f(X_\mathrm{new})\) (e.g. our predicted price for the new house)
- Our goal is for \(\hat Y_\mathrm{new} \approx Y_\mathrm{new}\): the true (but unknown) price of the new house.
Assumptions: what is random?
- Using the notation from last lecture, you’ll note that the training data \((X_i, Y_i)\) and the test data \((X_\mathrm{new}, Y_\mathrm{new})\) are random variables.
- This is a modelling choice. As statisticians, our primary tool for reasoning about the learning process is assuming something is random.
- In particular, we’re going to make the following assumptions about our data:
- There is some joint distribution \(\mathbb{P}_{X,Y}\) that governs the covariate/response pairs we see in the world
- Our training data are i.i.d. (independently and identically distributed) random samples from that distribution
- Our test data are also i.i.d. random samples from that distribution.
But isn’t our data given to us?
- In the real world, your training data will actually be some set of numbers. For example, your boss may hand you a CSV with covariate/response pairs. How are these quantities random, if the data are fixed and handed to us?
- Again, randomness is a modelling assumption that we use to simplify our lives. So how does this assumption help us?
- What we want to codify in our model: the data we make predictions on (our test data) are “sufficiently similar” to our training data
- How do we codify “sufficiently similar” in a probabilistic way?
- One way is to assume that they come from the same population distribution.
- To “come from the same population distribution” means that, mathematically, we treat them as random samples from \(\mathbb{P}_{X,Y}\).
Prediction versus Inference
In learning we are primarily concerned with prediction over inference.
- Inference: The goal is making a probabilistic statement about the relationship between \(X\) and \(Y\).
- (For example, we might want to know how much the price of a house increases for each additional bedroom.)
- Prediction: The goal is producing accurate \(\hat Y_\mathrm{new}\).
- We don’t really care about the underlying relationship, or even about making probabilistic statements about it.
- We just want some procedure to give us good predictions.
The Supervised Learning Procedure: Two Perspectives
CS/Algorithmic Perspective
Learning is often presented as an algorithm for producing a prediction rule \(\hat f\) from data \(\mathcal{D}\). This algorithm consists of 6 steps, which we will examine in the context of linear regression:
- Train/val/test split: Divide data into training and testing sets
- Given our dataset \(\mathcal{D}\), we split it into a training set \(\mathcal{D}_\mathrm{train}\), a validation set \(\mathcal{D}_\mathrm{val}\), and a test set \(\mathcal{D}_\mathrm{test}\).
- Define a Predictive Model: Select a set of candidate \(\hat f\) which could possibly predict \(Y\) from \(X\).
- For linear regression, the predictive model will be a linear function of the form \(\hat f(X) = X^\top \hat\beta\) for some \(\hat \beta\).
- Training: Define a training algorithm (a procedure to choose a \(\hat \beta\) from the training data) and apply it to \(\mathcal{D}_\mathrm{train}\).
- Most training algorithms involve minimizing a loss function over training data.
- For linear models, we often choose \(\hat f\) to minimize the squared error loss over training data. \[L(Y, \hat f(X)) = (Y - \hat f(X))^2, \qquad \hat f = \mathrm{argmin}_{\hat f} \frac{1}{n} \sum_{i=1}^{n} L(Y_i, \hat f(X_i)). \]
- Validation: Test performance on withheld validation data.
- We compute the average loss on the validation set to evaluate how well our model generalizes to unseen data.
- Iteration: Refine model based on evaluation results
- We may choose to change our hypothesis class, the set of features, the loss function, or the training algorithm to try to reduce validation error.
- Testing (confusingly referred to by some in the ML/AI world as “Inference”): Once satisfied with the model, evaluate on the test set to estimate performance and/or use this model to make predictions on new data.
Statistical Perspective
Over the next few lectures, we will derive the CS/algorithmic perspective from first statistical principles. For this lecture, we will focus on a statistical perspective on Steps 2, 3, and 6:
| Step | CS Perspective | Statistical Perspective |
|---|---|---|
| 2 | Predictive Model | Hypothesis Class |
| 3 | Training | Estimation |
| 6 | Testing | Prediction |
Before defining these components, however, we’ll introduce one concept that has no direct counterpart in the CS steps but underpins the statistical perspective: the statistical model, which we’ll define after taking a moment to think about our data.
Statistical Models
When thinking about learning from a statistical lens, before we even begin thinking about a predictive model, it is often best to define our statistical model, which is our best guess over the possible relationships between the covariates \(X\) and the response variable \(Y\)
- Formally, it is a set of conditional distributions over \(Y \mid X\) that could possibly generate the data we observe. (We’ll go through this idea in detail in a sec.)
- Informally, it tells us how the response \(Y\) is composed of deterministic signal from \(X\) and unexplainable random noise.
Because our goal is to predict \(Y\) from \(X\), we often don’t care about where \(X\) comes from. We can often assume that \(X\) is just “given” to us (i.e. we condition on it).
It’s best to illustrate this concept through an example:
The Linear Regression Model
From STAT 306, you may recall that we typically assume that the relationship between \(X\) and \(Y\) can be described as: \[Y = f(X) + \varepsilon, \qquad f(X) = X^\top \beta \]
- \(\beta\) is some (unknown) vector.
- \(\varepsilon\) is the random error term, which is typically i.i.d. \(\varepsilon \sim N(0, \sigma^2)\) for some \(\sigma^2 > 0\).
- (You may notice that we don’t have an intercept term \(\beta_0\) in this model. In this course, it’s often convenient to assume that \(X\) has an “all-ones” covariate, i.e. \(X = [1, X_1, X_2, \ldots, X_p]\), so that the intercept term \(\beta_0\) is included with the other \(\beta\) terms, i.e. \(X^\top \beta = \beta_0 + X_1 \beta_1 + \ldots + X_p \beta_p\).)
This relationship (\(Y = f(X) + \varepsilon\), \(f(X) = X^\top \beta\)) is a statistical model, and it is the one we will encounter most in the first part of this course. Let’s see how it matches our informal and formal definitions.
Informal Analysis
The linear statistical model assumes that \(Y\) can be explained through a combination of deterministic signal from \(X\) and unexplainable random noise.
- Signal – \(f(X)\): A portion of \(Y\) comes from \(f(X) = X^\top \beta\), which is a deterministic transform of \(X\).
- Noise – \(\varepsilon\): The other portion of \(Y\) comes from \(\varepsilon\), which is random and independent of \(X\). Thus, knowing \(X\) (and \(f(X)\)) is insufficient to perfectly predict \(Y\).
Formal Analysis
Let’s show how this linear statistical model is equivalent to a set of conditional distributions:
For any given \(\beta\), if we are given \(X = x\), then we have that \(Y \sim \mathcal{N}(x^\top\beta, \sigma^2)\). Thus, the corresponding statistical model/set of possible conditional distributions is:
\[\left\{ \mathbb{P}_{Y \mid X} \: : \: p_{Y \mid X}(y \mid x) \: = \: \mathcal{N}(y; \, x^\top\beta, \sigma^2), \:\: \beta \in \mathbb R^p \right\}.\]
We refer to \(\beta\) as the parameters of the model, as a given value of \(\beta\) specifies a particular distribution within the set.
Going back to the linear model \(Y = X^\top \beta + \varepsilon,\) which of these values are random?
Answer
\[ \underbrace{Y}_\text{random} = \underbrace{X^\top}_\text{random} \underbrace{\beta}_\text{fixed} + \underbrace{\varepsilon}_\text{random} \]
\(\varepsilon\) is random; note that above we are assuming that \(\varepsilon \sim N(0, \sigma^2)\)
- \(X\) is random; again we make the assumption that our data are random samples from a distribution (also it’s noted as a capital letter!)
- \(\beta\) is fixed! It defines the particular distribution in our statistical model that actually relates \(X\) and \(Y\)
- \(Y\) is also random; again by assumption, but also because \(Y\) equals some function involving random variables
Note that if we instead consider \(Y \mid X = x\), then
\[ \underbrace{Y \mid X=x}_\text{random} = \underbrace{x^\top}_\text{fixed} \underbrace{\beta}_\text{fixed} + \underbrace{\varepsilon}_\text{random} \]
Other Models
This linear regression model is just one statistical model we could use to describe our data. Almost any family of conditional distributions can be used instead. For example:
The more general linear regression model: rather than assuming the noise component \(\varepsilon\) is Gaussian (with some specified variance), we could instead assume any arbitrary (zero-mean) distribution over the noise
\[ Y = f(X) + \varepsilon, \qquad f(X) = X^\top \beta, \qquad \mathbb E[\varepsilon] = 0.\]
- This statistical model is equivalent to the set of conditional distributions \[\left\{ \mathbb{P}_{Y \mid X} \: : \: \mathbb E[Y \mid X = x] \: = \: x^\top\beta, \:\: \beta \in \mathbb R^p \right\}\]
- This model allows for the possibility of non-normal residuals.
The more general regression model: we could instead consider the slightly more general case where the expectation is an arbitrary function of \(X\) (not just a linear one):
\[ Y = f(X) + \varepsilon, \qquad f(X) \text{ is an arbitrary function}, \qquad \mathbb E[\varepsilon] = 0. \]
- This statistical model is equivalent to the set of conditional distributions \[\left\{ \mathbb{P}_{Y \mid X} \: : \: \mathbb E[Y \mid X = x] \: = \: f(x), \:\: f \text{ is an arbitrary function} \right\}\]
- Note again that our general linear model is a special case; however this model allows for more complex relationships between \(X\) and \(Y\).
- In general, this statistical model is too general to analyze properly. We’ll typically add some simplifying restrictions to \(f\), such as:
- \(f\) is a polynomial of degree \(d\), or
- \(f\) is continuous and differentiable everywhere.
Now that we have a statistical model, which tells us the general relationship between covariates and response, let’s dive through the learning procedure.
Hypothesis Class
Recall that our statistical model says that there’s some portion of our response that’s “learnable” from our covariates (\(X^\top \beta\) in our linear statistical model example) and some “unlearnable” portion of our response (\(\varepsilon\)). Our goal is to learn an approximation \(\hat f(X)\) to the learnable portion of our data; i.e.
\[ \hat f(X) \approx f(X). \]
Before we can go about choosing a specific \(\hat f\), we need to specify the set of functions \(\mathcal H\) that we are considering.
- \(\mathcal H\) is often referred to as the hypothesis class of our predictive model.
- Often, it’s good to match the functional form of our statistical model.
- In our running example, since we assume that \(f(X) = X^\top \beta\), it would make sense for \(\hat f(X) = X^\top \hat \beta\) for some \(\hat \beta\).
- However, later in this course we’ll discuss when it’s advantageous for \(\hat f\) to look nothing like our statistical model.
Putting it all together for our running example, it makes sense to define the hypothesis class \(\mathcal H\) as the set of all linear functions of \(X\):
\[ \mathcal H = \left\{ \hat f(\cdot) \: : \: \hat f(X) = X^\top \hat \beta, \qquad \hat \beta \in \mathbb R^p \right\}. \]
Estimation
Now that we’ve defined the set of \(\hat f\) that we’re considering, how do we go about choosing which \(\hat f \in \mathcal H\) is best at approximating \(f(X)\)? We use the data!
- The estimation step involves us choosing a specific function \(\hat f\) from the hypothesis class that best fits our training data.
- Alternatively, since all \(\hat f \in \mathcal H\) can be written as \(\hat f(X) = X^\top \hat \beta\), and \(f(X) = X^\top \beta\) under our linear statistical model, we could view the estimation step as choosing the best \(\hat \beta\) parameters to estimate the true parameters \(\beta\):
\[ \hat f(X) \approx f(X) \quad \Leftrightarrow \quad X^\top \hat\beta \approx X^\top \beta \quad \Leftrightarrow \quad \hat\beta \approx \beta. \]
Defining “Goodness” Through Loss Functions:
In order to choose the “best” \(\hat f \in \mathcal H\), we need to define what “best” means.
We do so by defining a loss function \(L(Y, \hat f(X))\), which quantifies how poorly \(\hat f\) is doing at a per-prediction level.
The most common loss for regression (and the one we’ll see most throughout this course) is the squared error loss:
\[ L(Y, \hat f(X)) = \left( Y - \hat f(X) \right)^2. \]
We’ll discuss classification losses in the next lecture.
Our Goal – Minimizing Expected Loss:
Once we have defined a loss function, the best possible \(\hat f\) is the one that minimizes the expected loss over all possible \((X, Y)\) pairs drawn from \(\mathbb P_{X,Y}\):
\[ f_\mathrm{best} = \mathrm{argmin}_{\hat f \in \mathcal H} \mathbb E[ L \left(Y, \hat f(X) \right)]. \]
Unfortunately, computing \(f_\mathrm{best}(X)\) would require exactly knowing \(\mathbb P_{X,Y}\) (or equivalently, having an infinite amount of data from this distribution). We only have access to \(n\) training pairs.
We therefore have to use procedures that approximate \(f_\mathrm{best}(X)\) from finite data.
There are many statistically rational estimation procedures. The most basic procedure is referred to as Empirical Risk Minimization (ERM). (We’ll talk about others throughout this course.)
The OLS/ERM Procedure:
If we have “enough” training data1 from \(\mathbb P_{X,Y}\), then – by the law of large numbers2 – we can assume that:
\[ \mathbb E[ L \left(Y, \hat f(X) \right)] \approx \underbrace{\frac{1}{n} \sum_{i=1}^n L(Y_i, \hat f(X_i))}_{=: \hat R_{\hat f}} \qquad \forall \hat f \in \mathcal H. \]
We refer to \(\hat R_{\hat f}\) as the empirical risk associated with \(\hat f\). It is the average loss on the training data.
Empirical risk minimization thus aims to approximate \(\hat f_\mathrm{best}(X)\) by choosing the \(\hat f\) with the lowest empirical risk (average training data loss).
\[\hat{f}_\mathrm{ERM} = \mathrm{argmin}_{\hat f \in \mathcal H} \hat R_{\hat f} = \mathrm{argmin}_{\hat f \in \mathcal H} \frac{1}{n}\sum_{i=1}^n L \left(Y_i, \hat f(X_i)\right).\]
Applying ERM to our Linear Hypothesis Class/Squared Loss
Going back to our running example (linear hypothesis class, squared loss), and recalling that \(\hat f\) is specified through \(\hat \beta\), we have that
\[\hat{\beta}_\mathrm{ERM} = \mathrm{argmin}_{\hat \beta} \frac{1}{n}\sum_{i=1}^n \left(Y_i - X_i^\top \hat \beta\right)^2. \]
Taking the derivative and setting to zero:
\[\frac{\partial}{\partial \beta}\sum_{i=1}^n (Y_i - X_i^\top\hat\beta)^2 = -2\sum_{i=1}^n X_i(Y_i - X_i^\top\hat\beta) = 0\]
By arranging all of our training data into a matrix \(\boldsymbol X \in \mathbb{R}^{n \times p}\) and a vector \(\boldsymbol Y \in \mathbb{R}^n\), with
\[\boldsymbol X = \begin{bmatrix} -X_1^\top- \\ -X_2^\top- \\ \vdots \\ -X_n^\top- \end{bmatrix} \in \mathbb{R}^{n \times p}, \qquad \boldsymbol Y = \begin{bmatrix} Y_1 \\ Y_2 \\ \vdots \\ Y_n \end{bmatrix},\]
we get:
\[\hat{\beta}_\mathrm{ERM} = (\boldsymbol X^\top \boldsymbol X)^{-1} \boldsymbol X^\top \boldsymbol Y.\]
You should recognize \(\hat{\beta}_\mathrm{ERM}\) as the ordinary least squares (OLS) estimator for linear models from STAT 306!
From STAT 306, you may have learned that we want to choose \(\hat \beta\) through the principle of maximum likelihood estimation. It turns out that, in the case of our linear statistical model, it yields the same \(\hat \beta\) as ERM!
Goal of maximum likelihood estimation: Find the parameters that maximize the likelihood of observing our data.
- The likelihood function maps a distribution from our model (parameterized by \(\beta\)) to the probability density of our observed training data.
Likelihood function for linear regression.
For the linear regression model with normal errors, the likelihood function is: \[\mathcal{L} (\beta, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(Y_i - X_i^\top\beta)^2}{2\sigma^2}\right)\]
The term inside the product is the density of the normal distribution \(\mathcal{N}(X_i^\top\beta, \sigma^2)\) evaluated at \(Y_i\).
We take the product over all \(n\) training examples \((X_i, Y_i)\) because we assume that the data points are i.i.d.
We choose \(\hat\beta\) to maximize the likelihood function, i.e. the \(\hat\beta\) that make observed data most probable.
Log-likelihoods are easier to work with.
Taking the log of the likelihood function, log-likelihood: \[\ell(\beta, \sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (Y_i - X_i^\top\beta)^2\]
The logarithm is a monotonic function, so maximizing the likelihood is equivalent to maximizing the log-likelihood.
The maximizer of this log likelihood function (and thus the maximum of the likelihood function) is thus equal to: \[\mathrm{argmax}_{\beta} -\sum_{i=1}^n (Y_i - X_i^\top\beta)^2 = \mathrm{argmin}_{\beta} \sum_{i=1}^n (Y_i - X_i^\top\beta)^2,\]
Details:
The \(-\frac{n}{2}\log(2\pi\sigma^2)\) constant and the \(\frac{1}{2\sigma^2}\) scaling term also don’t affect the maximum, so we can ignore them.
This is exactly the same optimization problem we derived for ERM! Thus, \[ \hat{\beta}_\mathrm{MLE} = (\boldsymbol X^\top \boldsymbol X)^{-1} \boldsymbol X^\top \boldsymbol Y = \hat{\beta}_\mathrm{ERM} = \hat{\beta}_\mathrm{OLS} \]
Key Insight: For linear regression with normal errors, MLE and OLS/ERM give identical results! However, if we had used a different loss function (e.g. \(L(Y_i, \hat f(X_i)) = |Y_i - \hat f(X_i)|\)), the ERM estimator would not be the same as the OLS/MLE estimator.
Prediction
Now that we have fit an estimator \(\hat f(X) = X^\top \hat\beta_\mathrm{ERM} \approx X^\top \beta\), we can use this function to make predictions about the responses for new data.
- Given a new set of covariates \(X_\mathrm{new}\) we can make the following prediction about its associated response:
\[ \hat Y_\mathrm{new} = \hat f(X_\mathrm{new}) = X^\top_\mathrm{new} \hat\beta_\mathrm{ERM} \approx Y_\mathrm{new}. \]
- Depending on the learning problem (i.e. our choice of statistical model, hypothesis class, and loss function) we may end up with different prediction rules. We’ll discuss how to derive prediction rules in the next lecture.
Summary
This lecture introduced the statistical framework for learning:
- Statistical models define the probabilistic structure of our data
- Hypothesis classes define the functions of covariates \(X\) that we consider to approximate the “signal”/“learnable” portion of our response \(Y\)
- Estimation finds the best parameters using ERM
- Prediction uses fitted models to make predictions on new data
- This framework will extend to more complex models and methods throughout the course.
- Linear regression serves as our foundational example, where the ERM produces the OLS estimator from STAT 306.
- In the next lecture, we will apply this statistical framework to classification problems.