Lecture 1: Probability and Linear Algebra Review

Author

Geoff Pleiss

Published

July 7, 2026

Learning Objectives

By the end of this lecture, you will be able to:

  1. Define a random variable and understand when to use them to model quantities
  2. Derive probability rules through the product and sum rules
  3. Apply linearity of expectation and the law of total expectation to simplify calculations
  4. Apply the SVD decomposition and describe the properties of each component matrix

Probability Review

Random Variables

Motivation

  • Example: Let’s say you want to grab a coffee at Loafe and you want to know how long you’ll have to wait in line.
  • Denote this time by the variable \(A\)
  • \(A\) depends on a multitude of factors:
    • How hot it is outside
    • What day of the week it is
    • How late Josh and his friends were up playing video games, thus leading them to take up spots in line
  • While we could try to model all of these factors, it would be infeasible to do so.
  • Instead, we can treat \(A\) as a random variable: a variable whose value is randomly sampled from some distribution.

Notation

  • We (almost) always denote random variables with uppercase letters (e.g., \(A\))
  • We (almost) always denote their realizations (i.e., specific values they can take) with lowercase letters (e.g., \(a\)).

Joint Random Variables

  • Throughout this class, most of the probability we will encounter will be concerned with relationships between two or more random variables.
  • Example: maybe we want to understand how the temperature outside, denoted by \(B\), affects the Loafe line length.
  • Again, \(B\) depends on many factors:
    • What time of year it is
    • Whether or not it’s sunny outside
    • How many flights Sarah took last year, thus leading to an increase in greenhouse gases
  • We can treat \(B\) as a random variable as well.
  • \(A\) and \(B\) are related to one another, potentially in a causal manner. If we treat them as joint random variables, we can derive many useful probabilistic representations about their relationship.

Distributions and The Two Rules of Probability

  • Given two random variables (A) and (B), we can describe this relationship through a joint probability distribution, which can be specified by a PMF (for discrete random variables) or a density (for continuous random variables):

    \[ \begin{cases} p_{A,B}(a, b) = \mathbb P(A=a, B=b) & \text{PMF for discrete random variables} \\ p_{A,B}(a, b) & \text{density for continuous random variables} \end{cases} \]

WarningWarning! Notation Overload for PMFs/Densities

Here, and throughout the rest of the course, we will use the notation \(p_{A,B}\) for both PMFs of discrete random variables as well as densities for continuous random variables.

  • We can also describe (A) and (B) through:
    • Conditional distributions, i.e., \(p_{A|B}(a|b)\) or \(p_{B|A}(b|a)\)
    • Marginal distributions, i.e., \(p_A(a)\) or \(p_B(b)\)
  • While there are many fundamental rules of probability to manipulate these distributions, most of them can be derived from two basic rules: the product rule and the sum rule.

(All the rules that follow will be defined/derived for continuous random variables, but they can be applied to discrete random variables by substituting “PMF” for “density” and replacing the integrals with sums.)

The Product Rule

The product rule allows us to decompose a joint density over two continuous random variables into the product of a conditional and marginal density:

\[\begin{align*} p_{A,B}(a, b) &= p_{A|B}(a|b) \, p_B(b) \\ &= p_{B|A}(b|a) \, p_A(a) \end{align*}\]

  • This rule can be applied recursively in the case of more than two continuous random variables.
  • This rule gives rise to lots of useful facts from probability theory.

Independence

  • We say that \(A\) and \(B\) are independent if \(p_{A,B}(a,b) = p_A(a) \, p_B(b)\); that is, their joint density is the product of their marginal densities.

  • By the product rule, for independent random variables we have that

    \[ p_A(a) \, p_B(b) = p_{A,B}(a,b) = p_{A|B}(a|b) \, p_B(b), \]

    and, through some algebra, that \(p_A(a) = p_{A|B}(a|b)\). (Similarly, \(p_B(b) = p_{B|A}(b|a)\).)

  • In other words, when \(A\) and \(B\) are independent, the occurrence of \(B\) does not affect the probability of \(A\), and vice versa.

Bayes’ Rule

We can derive Bayes’ formula

\[ p_{B|A}(b|a) = \frac{p_{A|B}(a|b) \, p_B(b)}{p_A(a)} \]

using the product rule by starting from the identity \(p_{B,A}(b,a) = p_{A,B}(a,b)\) and simplifying.

The Sum Rule

The sum rule allows us to obtain a marginal density for \(A\) (or \(B\)) from a joint density over \(A\) and \(B\):

\[ p_A(a) = \int_{b} p_{A,B}(a, b) \: \mathrm{d}b. \]

  • Again, this rule can be extended to three or more variables recursively.
  • This rule is instrumental in establishing properties about expectations:

Linearity of Expectation

  • We define the expected value of \(A\) as:

    \[ \mathbb{E}[A] := \int_{a} a \: p_A(a) \: \mathrm{d}a. \]

  • Similarly, the expected value of some function of \(A\) and \(B\) is defined as:

    \[ \mathbb{E}[g(A, B)] := \int_a \int_b g(a, b) \, p_{A,B}(a, b) \: \mathrm{d}b \: \mathrm{d}a. \]

  • We can use the sum rule in conjunction with Fubini’s theorem to derive one of the most important formulas in all of probability:

    \[ \mathbb E[A + B] = \mathbb E[A] + \mathbb E[B] \]

\[\begin{align*} \mathbb{E}[A + B] &= \int_a \int_b (a + b) \, p_{A,B}(a, b) \: \mathrm{d}b \: \mathrm{d}a \\ &= \int_a \int_b a \, p_{A,B}(a, b) \: \mathrm{d}b \: \mathrm{d}a \\ &\quad + \int_a \int_b b \, p_{A,B}(a, b) \: \mathrm{d}b \: \mathrm{d}a \\ &= \int_a a \int_b p_{A,B}(a, b) \: \mathrm{d}b \: \mathrm{d}a \\ &\quad + \int_b b \int_a p_{A,B}(a, b) \: \mathrm{d}a \: \mathrm{d}b \\ &= \int_a a \: p_A(a) \: \mathrm{d}a + \int_b b \: p_B(b) \: \mathrm{d}b \\ &= \mathbb{E}[A] + \mathbb{E}[B] \end{align*}\]

This formula, known as linearity of expectation, holds even when \(A\) and \(B\) are not independent! We will use this fact constantly throughout the course.

Conditional Expectations and The Tower Rule

  • The conditional expectation of \(A\) given \(B=b\) is defined as:

    \[ \mathbb{E}[A \mid B = b] := \int_{a} a \: p_{A|B}(a|b) \: \mathrm{d}a. \]

    It is the average value that \(A\) takes when we have the additional information that the random variable \(B\) takes on the value \(b\).

  • Note that this conditional expectation is a function of \(b\); i.e. we can write

    \[ \mathbb{E}[A \mid B = b] =: g(b).\]

  • The expression \(\mathbb{E} [A \mid B]\) should then be read as:

    \[ \mathbb{E}[A \mid B] = g(B). \]

    I.e. that we are applying the conditional expectation function to the random variable \(B\) rather than to a specific realization \(b\).

  • To relate \(\mathbb E[A \mid B]\) to \(\mathbb E[A]\), we can use the sum and product rules to get:

    \[ \mathbb E[A] = \mathbb E[ \mathbb E[ A \mid B]] \]

\[\begin{align*} \mathbb{E}[A] &= \int_a a \int_b p_{A,B}(a, b) \: \mathrm{d}b \: \mathrm{d}a \\ &= \int_a a \int_b p_{A|B}(a|b) \, p_B(b) \: \mathrm{d}b \: \mathrm{d}a \\ &= \int_b \left( \int_a a \: p_{A|B}(a|b) \: \mathrm{d}a \right) p_B(b) \: \mathrm{d}b \\ &= \int_b \mathbb{E}[A \mid B=b] \, p_B(b) \: \mathrm{d}b \\ &= \mathbb{E} \left[ \mathbb{E}[ A \mid B ] \right] \end{align*}\]

  • This rule is known as the Tower Rule. It allows us to relate marginal expectations and conditional expectations. It will also be very important in this course.

  • This notation is often confusing and scary. Try translating it back into probabilities via the sum and product rules, and you’ll fluently understand it in no time!

ImportantImportant: What is Random?

  • A (standard) expectation \(\mathbb E[A]\) is not a random variable (despite the fact that there’s a random variable inside the expectation).

    Why?

    If we go back to the definition: \(\mathbb E[A] = \int a \, p_A(a) \, da\), note that the two terms in the integral, \(a\) and \(p_A(a)\), are functions of \(a\), which is a realized quantity and therefore not random. So we’re integrating non-random quantities together, giving us a non-random output.

  • The conditional expectation \(\mathbb E[A \mid B=b]\) is also not a random variable.

    Why?

    Again going back to the definition: \(\mathbb E[A \mid B=b] = \int a \, p_{A|B}(a|b) \, da\), the terms inside the integral are functions of \(a\) and \(b\) (realized quantities, not random variables).

  • However, the conditional expectation \(\mathbb E[A \mid B]\) is a random variable.

    Why?

    Recall that we can view \(\mathbb E[A \mid B=b]\) as some deterministic function of \(b\); i.e. \(\mathbb E[A \mid B=b] = g(b)\). When we plug a realization (i.e. not a random variable) into \(g\), the output is a fixed (not random) quantity.

    However, when we plug in a random variable into \(g\), the output is a random quantity! So \(\mathbb E[A \mid B] = g(B)\) is a random variable due to the randomness in \(B\).

Linear Algebra Review

We’re not going to go into too much review about linear algebra. Based on the course prerequisites, I’m assuming that you’re very familiar with the following concepts:

  • Matrix multiplication
  • Matrix inverses
  • Matrix rank
  • The trace and determinant of a matrix
  • Orthogonal, symmetric, and positive definite matrices
  • Vector inner product
  • Vector norms

If you are unfamiliar or uncomfortable with any of these ideas, brush up on them quickly! We’ll make use of them very extensively starting with the next lecture.

One concept that you hopefully have seen before (but maybe haven’t used too much) is the Singular Value Decomposition (or SVD). It is an important tool for understanding the intuitions behind many of the methods we will discuss in this course.

Notation

Matrices will (almost) always be represented by bold uppercase letters (e.g. \(\boldsymbol A\)), regardless of whether they’re random or deterministic.

The Singular Value Decomposition

Any matrix \(\boldsymbol M \in \mathbb R^{n \times p}\) can be decomposed as \(\boldsymbol M = \boldsymbol U \boldsymbol D \boldsymbol V^\top\). Writing \(r = \min(n, p)\) for the smaller of the two dimensions:

  • \(\boldsymbol U \in \mathbb{R}^{n \times r}\) has orthonormal columns: they are mutually orthogonal unit vectors, so \(\boldsymbol U^\top \boldsymbol U = \boldsymbol I\).
  • \(\boldsymbol D \in \mathbb{R}^{r \times r}\) is square and diagonal: all off-diagonal entries are zero, and the diagonal entries (the singular values, denoted by \(d_1, d_2, \ldots, d_r\)) are non-negative.
  • \(\boldsymbol V \in \mathbb{R}^{p \times r}\) has orthonormal columns (\(\boldsymbol V^\top \boldsymbol V = \boldsymbol I\)), so \(\boldsymbol V^\top \in \mathbb{R}^{r \times p}\).

This is the thin (or reduced) SVD. We use it throughout the course because the square, diagonal \(\boldsymbol D\) is convenient to work with (e.g. it is easy to invert when analyzing ridge regression). When \(\boldsymbol M\) is square (\(n = p\), as in the 2D example below), \(\boldsymbol U\) and \(\boldsymbol V\) are themselves square orthonormal matrices — i.e. rotation matrices.

This decomposition has so many uses in statistics and beyond, and we’ll see it many times throughout this class.

You may have previously seen the full SVD, which pads the thin version with extra orthonormal columns so that \(\boldsymbol U\) and \(\boldsymbol V\) become square: \[\boldsymbol M = \boldsymbol U \boldsymbol D \boldsymbol V^\top, \qquad \boldsymbol U \in \mathbb{R}^{n \times n}, \quad \boldsymbol D \in \mathbb{R}^{n \times p}, \quad \boldsymbol V \in \mathbb{R}^{p \times p}.\] Now \(\boldsymbol U\) and \(\boldsymbol V\) are square orthogonal matrices (both their rows and columns are orthonormal: \(\boldsymbol U^\top \boldsymbol U = \boldsymbol U \boldsymbol U^\top = \boldsymbol I\), and likewise for \(\boldsymbol V\)), while \(\boldsymbol D\) becomes a rectangular “diagonal” matrix:

  • If \(n > p\) (more rows than columns): \(\boldsymbol D\) holds the \(p\) singular values in its top \(p \times p\) block, with the remaining \(n - p\) rows all zero.
  • If \(n < p\) (more columns than rows): \(\boldsymbol D\) holds the \(n\) singular values in its leftmost \(n \times n\) block, with the remaining \(p - n\) columns all zero.

Those zero rows (or columns) multiply the “extra” columns of \(\boldsymbol U\) (or \(\boldsymbol V\)), so they contribute nothing to \(\boldsymbol M\). Dropping them recovers the thin SVD above.

Intuition

To understand its use, it’s important to first understand the information that’s captured in each of the terms in this decomposition. And, because everything is easier to visualize in 2D, let’s assume that \(n = p = 2\).

SVD Intuition (adapted from Wikipedia)

Any matrix \(\boldsymbol M\) can be thought of as a (linear) operation to transform a vector (e.g. by rotation, scale, etc.).

  • Consider the vectors \(\boldsymbol o\) and \(\boldsymbol r\), visualized by the orange and red arrows (respectively) in the upper left.
  • Applying the matrix to these vectors produces \(\boldsymbol M \boldsymbol o\) and \(\boldsymbol M \boldsymbol r\), depicted by the arrows in the upper right.
  • Note how these arrows have been rotated and stretched.

The SVD breaks this transformation down into three composable steps:

  • First, we rotate the vectors with the matrix \(\boldsymbol V^\top\) (producing \(\boldsymbol V^\top \boldsymbol o\) and \(\boldsymbol V^\top \boldsymbol r\)). Recall that orthonormal matrices are rotation matrices, where multiplying them against a vector is the same as rotating that vector about the origin.
  • Next, we stretch the vectors along the principal axis (x- and y-axis) with the matrix \(\boldsymbol D\) (producing \(\boldsymbol D \boldsymbol V^\top \boldsymbol o\) and \(\boldsymbol D \boldsymbol V^\top \boldsymbol r\)). Recall that diagonal matrices shrink or stretch vectors along the principal axes.
  • Finally, we apply one more rotation to the vectors with the matrix \(\boldsymbol U\) (producing \(\boldsymbol U \boldsymbol D \boldsymbol V^\top \boldsymbol o = \boldsymbol M \boldsymbol o\) and \(\boldsymbol U \boldsymbol D \boldsymbol V^\top \boldsymbol r = \boldsymbol M \boldsymbol r\)).

Amazingly, any linear transformation can be decomposed into this rotate-stretch-rotate pattern!

How We’ll Use the SVD

The most important information in the SVD lives in the \(\boldsymbol D\) matrix. It can tell us the effective dimensionality of the matrix.

  • In the example above, note how multiplying against \(\boldsymbol D\) stretched the x-axis but shrunk the y-axis.
  • This implies that \(d_1\) (corresponding to the x axis) is a very large number, while \(d_2\) (corresponding to the y axis) is a very small number.
  • Now imagine if \(d_1 \gg d_2\), with \(d_2 \approx 0\):
    • Any vector \(\boldsymbol D \boldsymbol V^\top \boldsymbol z\) would then approximately live on the x axis.
    • Even after rotating with \(\boldsymbol U\), most vectors \(\boldsymbol M \boldsymbol z = \boldsymbol U \boldsymbol D \boldsymbol V^\top \boldsymbol z\) would approximately live on a line.
    • This implies that \(\boldsymbol M\) effectively collapses all points to nearly live on a single line. So it collapses 2D points to a nearly 1D subspace.
  • Conversely, if \(d_1 \approx d_2\), then no dimension gets collapsed more than the other. \(\boldsymbol D\) isn’t doing too much (assuming without loss of generality that \(d_1 \approx d_2 \approx 1\)), so \(\boldsymbol M\) is mostly characterized by the two rotation matrices \(\boldsymbol U\) and \(\boldsymbol V\).
NoteTakeaway

The singular values tell us how many dominant dimensions a matrix has. In 2D:

  • If \(d_1 \approx d_2 \approx 1\), then the matrix isn’t doing much more than rotating vectors.
  • If \(d_1 \gg d_2\), then the matrix will collapse vectors to a nearly 1D subspace.

Conclusion

  • We will use (jointly-distributed) random variables to model data, models that depend on data, and predictions that depend on models that depend on data.
  • You will need to manipulate marginal, joint, and conditional probabilities and expectations of these random variables throughout the course.
  • This review has covered most of the probability rules that we’ll use, but just remember that you can always derive any of them through the product and sum rules!
  • The SVD tells us the effective dimensionality of a matrix. We’ll use it lots when analyzing methods throughout this course!