Expected values, Variance, Correlation, and Generating functions
Last modified — 16 Mar 2026
Given a random variable \(X\),
\(\mathbb{E}[g(X)]\) is the weighted average value of \(g(X)\)
where the weights are given by the probabilities of \(X\) taking on different values.
In primary school, you learned about sample averages of \(n\) data points.
If we were to construct a discrete random variable \(Y\) by giving each observed value the probability of \(1/n\),
then \(\mathbb{E}[Y] = \frac{1}{n}\sum_{i=1}^n y_i\) would be the sample average of the observed values.
Let \(X \sim {\mathrm{Binom}}(n, \theta)\). We want to compute \(\mathbb{E}[X]\).
We have that \[\begin{aligned} \mathbb{E}[X] &= \sum_{x=0}^n x \binom{n}{x} \theta^x (1-\theta)^{n-x} = \sum_{x=1}^n x \binom{n}{x} \theta^x (1-\theta)^{n-x} && \text{sum is 0 when } x=0\\ &= \sum_{x=1}^n n \binom{n-1}{x-1} \theta^x (1-\theta)^{n-x} && \text{because } x\binom{n}{x} = n\binom{n-1}{x-1}\\ &= n \sum_{y=0}^{n-1} \binom{n-1}{y} \theta^{y+1} (1-\theta)^{(n-1)-y} && \text{substitute } x = y + 1 \\ &= n\theta \sum_{y=0}^{n-1} \binom{n-1}{y} \theta^y (1-\theta)^{(n-1)-y} && \text{compare to } {\mathrm{Binom}}(n-1, \theta)\\ &= n\theta. \end{aligned}\]
Let \(X \sim \textrm{Gamma}(\alpha, \lambda )\). We want to compute \(\mathbb{E}[X]\).
We have that \[\begin{aligned} \mathbb{E}[X] &= \int_0^\infty x \frac{\lambda^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\lambda x} \mathsf{d}x \\ &= \frac{\lambda^\alpha}{\Gamma(\alpha)} \int_0^\infty x^{\alpha} e^{-\lambda x} \mathsf{d}x \\ &= \frac{\lambda^\alpha}{\Gamma(\alpha)} \cdot \frac{\Gamma(\alpha + 1)}{\lambda^{\alpha + 1}} \int_0^\infty \frac{\lambda^{\alpha+1}}{\Gamma(\alpha+1)} x^{\alpha + 1- 1} e^{-\lambda x} \mathsf{d}x \\ &= \frac{\Gamma(\alpha + 1)}{\lambda \Gamma(\alpha)} && \text{integrates to 1 because Gamma}(\alpha + 1, \lambda)\\ &= \frac{\alpha \Gamma(\alpha)}{\lambda \Gamma(\alpha)} && \text{because } \Gamma(\alpha + 1) = \alpha \Gamma(\alpha) \\ &= \frac{\alpha}{\lambda}. \end{aligned}\]
Let \(X \sim {\mathrm{Gam}}(\alpha, \lambda)\) where \(\alpha>0\) and \(\lambda > 0\). Recall that the PDF of a RV \(Y\sim{\mathrm{Gam}}(\theta, \beta)\) is given by \[f_X(x) = \frac{\beta^\theta}{\Gamma(\theta)} x^{\theta - 1} e^{-\beta x} I_{(0,\infty)}(x).\]
(Where the expected value exists.)
For the last property, the converse is false.
Find \(\mathbb{E}\left[\frac{1}{2}(X + Y)^2\right]\).
If \(X\) and \(Y\) are both discrete random variables, then \[\begin{aligned} \mathbb{E}[g(X, Y)] &= \sum_{x} \sum_{y} g(x, y) p_{X,Y}(x, y). \end{aligned}\] If \(X\) and \(Y\) are jointly absolutely continuous random variables, then \[\begin{aligned} \mathbb{E}[g(X, Y)] &= \int_{-\infty}^\infty \int_{-\infty}^\infty g(x, y) f_{X,Y}(x, y) \mathsf{d}x \mathsf{d}y. \end{aligned}\]
\[\begin{aligned} \mathbb{E}[g(X)h(Y)] &= \int_{-\infty}^\infty \int_{-\infty} ^\infty g(x) h(y) f_{X,Y}(x, y) \mathsf{d}x \mathsf{d}y \\ &= \int_{-\infty}^\infty \int_{-\infty}^\infty g(x) h(y) f_X(x) f_Y(y) \mathsf{d}x\mathsf{d}y && \text{$X$ and $Y$ are independent} \\ &= \int_{-\infty}^\infty g(x) f_X(x) \mathsf{d}x \int_{-\infty}^\infty h(y) f_Y(y) \mathsf{d}y \\ &= \mathbb{E}[g(X)] \mathbb{E}[h(Y)]. \end{aligned}\]
Let \(X\) and \(Y\) have joint PDF \[f_{X,Y}(x,y) = 8xyI_{\{0 < x < y < 1\}}(x,y).\]
(Where the variance exists.)
Tip
\[\begin{aligned} \operatorname{Var}(X) &= \mathbb{E}[(X - \mathbb{E}[X])^2] \\ &= \mathbb{E}[X^2 - 2X\mathbb{E}[X] + \mathbb{E}[X]^2] \\ &= \mathbb{E}[X^2] - 2\mathbb{E}[X]\mathbb{E}[X] + \mathbb{E}[X]^2 \\ &= \mathbb{E}[X^2] - \mathbb{E}[X]^2.\\ \Longrightarrow \operatorname{Var}(X) &\leq \mathbb{E}[X^2]. \end{aligned}\]
Hints: remember that \(\mathbb{E}[X] = 1/\lambda\) and that \(\Gamma(z) = (z-1)!\) for integer \(z > 1\).
If we have two random variables \(X\) and \(Y\), we can measure the relationship between them.
We can compute the covariance using the joint PMF/PDF of \(X\) and \(Y\).: \[\begin{aligned} \operatorname{Cov}(X, Y) &= \sum_{x} \sum_{y} (x - \mathbb{E}[X])(y - \mathbb{E}[Y]) p_{X,Y}(x, y).\\ \operatorname{Cov}(X, Y) &= \int_{-\infty}^\infty \int_{-\infty}^\infty (x - \mathbb{E}[X])(y - \mathbb{E}[Y]) f_{X,Y}(x, y) \, \mathsf{d}x \, \mathsf{d}y. \end{aligned}\]
Hint: If \(Z\sim {\mathrm{Binom}}(n, \theta)\), then \(\operatorname{Var}(Z) = n\theta(1-\theta)\).
Let \(X\) and \(Y\) be random variables with finite variances.
Covariance is not a standardized measure of the relationship between \(X\) and \(Y\).
For example, if we multiply \(X\) by 100, then \(\operatorname{Cov}(X, Y)\) will also be multiplied by 100.
Returning to the heads or tails example, we have that \[\begin{aligned} \rho_{XY} &= \frac{\operatorname{Cov}(X, Y)}{\sigma_X \sigma_Y} = \frac{-5/4}{\sqrt{5/4} \sqrt{5/4}} = -1. \end{aligned}\]
End of material for Midterm 2.
Stat 302 - Winter 2025/26