If we knew how to rotate our data, then we could more easily retain the structure.
PCA gives us exactly this rotation
PCA works when the data can be represented (in a lower dimension) as lines (or planes, or hyperplanes).
So, in two dimensions:
PCA reduced
Here, we can capture a lot of the variation and underlying structure with just 1 dimension,
instead of the original 2 (the colouring is for visualizing).
PCA bad
What about other data structures? Again in two dimensions
PCA bad
Here, we have failed miserably.
There is actually only 1 dimension to this data (imagine walking up the spiral going from purple to yellow).
However, when we write it as 1 PCA dimension, all the points are all “mixed up”.
Explanation
PCA wants to minimize distances (equivalently maximize variance).
This means it slices through the data at the meatiest point, and then the next one, and so on.
If the data are curved this is going to induce artifacts.
PCA also looks at things as being close if they are near each other in a Euclidean sense.
On the spiral, our intuition says that things are close only if the distance is constrained to go along the curve.
In other words, purple and blue are close, blue and yellow are not.
Kernel PCA
Classical PCA comes from \(\X= \U\D\V^{\top}\), the SVD of the (centered) data
However, we can just as easily get it from the outer product \(\mathbf{K} = \X\X^{\top} = \U\D^2\U^{\top}\)
The intuition behind KPCA is that \(\mathbf{K}\) is an expansion into a kernel space, where \[\mathbf{K}_{i,i'} = k(x_i,\ x_{i'}) = \langle x_i,x_{i'} \rangle\]
We saw this trick before with feature expansion.
Procedure
Specify a kernel function \(k\)
many people use \(k(x,x') = \exp\left( -d(x, x')/\gamma\right)\) where \(d(x,x') = \norm{x-x'}_2^2\)
Form \(\mathbf{K}_{i,i'} = k(x_i,x_{i'})\)
Double center \(\mathbf{K} = \mathbf{PKP}\) where \(\mathbf{P} = \mathbf{I}_n - \mathbf{11}^\top / n\)
Take eigendecomposition \(\mathbf{K} = \U\D^2\U^{\top}\)
The scores are still \(\mathbf{Z} = \U_M\D_M\)
Note
We don’t explicitly generate the feature map \(\longrightarrow\) there are NO loadings
An alternate view
To get the first PC in classical PCA, we solve \[\max_\alpha \Var{\X\alpha} \quad \textrm{ subject to } \quad \left|\left| \alpha \right|\right|_2^2 = 1\]
In the kernel setting we solve \[\max_{g \in \mathcal{H}_k} \Var{g(X)} \quad \textrm{ subject to } \quad\left|\left| g \right|\right|_{\mathcal{H}_k} = 1\]
Here \(\mathcal{H}_k\) is a function space determined by \(k(x,x')\).
Example kernels
\(k(x,x') = x^\top x'\)
gives back regular PCA
\(k(x,x') = (1+x^\top x')^d\)
gives a function space which contains all \(d^{th}\)-order
polynomials.
\(k(x,x') = \exp(-\norm{x-x'}_2^2/\gamma)\)
gives a function space spanned by the infinite Fourier basis
