Start simple
For any \(f : \R \rightarrow [0,1]\)
\[f(x) = f(x_0) + f'(x_0)(x-x_0) + \frac{1}{2}f''(x_0)(x-x_0)^2 + \frac{1}{3!}f'''(x_0)(x-x_0)^3 + R_3(x-x_0)\]
So we can linearly regress \(y_i = f(x_i)\) on the polynomials.
The more terms we use, the smaller \(R\).
Same thing, different orders
Code
arcuate %>%
ggplot(aes(position, fa)) +
geom_point(color = blue) +
geom_smooth(aes(color = "a"), formula = y ~ poly(x, 4), method = "lm", se = FALSE) +
geom_smooth(aes(color = "b"), formula = y ~ poly(x, 7), method = "lm", se = FALSE) +
geom_smooth(aes(color = "c"), formula = y ~ poly(x, 25), method = "lm", se = FALSE) +
scale_color_manual(name = "Taylor order",
values = c(green, red, orange), labels = c("4 terms", "7 terms", "25 terms"))
Code
library(cowplot)
g1 <- ggplot(tibble(cvscores, degrees = seq(max_deg)), aes(degrees, cvscores)) +
geom_point(colour = blue) +
geom_line(colour = blue) +
labs(ylab = 'LOO-CV', xlab = 'polynomial degree') +
geom_vline(xintercept = which.min(cvscores), linetype = "dotted")
g2 <- ggplot(arcuate, aes(position, fa)) +
geom_point(colour = blue) +
geom_smooth(
colour = orange,
formula = y ~ poly(x, which.min(cvscores)),
method = "lm",
se = FALSE
)
plot_grid(g1, g2, ncol = 2)
Other bases
- Polynomials
- \(x \mapsto \left(1,\ x,\ x^2, \ldots, x^p\right)\) (technically, not quite this, they are orthogonalized)
- Linear splines
- \(x \mapsto \bigg(1,\ x,\ (x-k_1)_+,\ (x-k_2)_+,\ldots, (x-k_p)_+\bigg)\) for some \(\{k_1,\ldots,k_p\}\)
- Fourier series
- \(x \mapsto \bigg(1,\ \cos(2\pi x),\ \sin(2\pi x),\ \cos(2\pi 2 x),\ \sin(2\pi 2 x), \ldots, \cos(2\pi p x),\ \sin(2\pi p x)\bigg)\)
Code
library(cowplot)
library(ggplot2)
relu_shifted <- function(x, shift) {pmax(0, x - shift)}
# Create a sequence of x values
x_vals <- seq(-3, 3, length.out = 1000)
# Create a data frame with all the shifted functions
data <- data.frame(
x = rep(x_vals, 5),
polynomial = c(x_vals, x_vals^2, x_vals^3, x_vals^4, x_vals^5),
linear.splines = c(relu_shifted(x_vals, 2), relu_shifted(x_vals, 1), relu_shifted(x_vals, 0), relu_shifted(x_vals, -1), relu_shifted(x_vals, -2)),
fourier = c(cos(pi / 2 * x_vals), sin(pi / 2 * x_vals), cos(pi / 4 * x_vals), sin(pi / 4 * x_vals), cos(pi * x_vals)),
function_label = rep(c("f1", "f2", "f3", "f4", "f5"), each = length(x_vals))
)
# Plot using ggplot2
g1 <- ggplot(data, aes(x = x, y = polynomial, color = function_label)) +
geom_line(size = 1, show.legend=FALSE) +
theme(axis.text.y=element_blank())
g2 <- ggplot(data, aes(x = x, y = linear.splines, color = function_label)) +
geom_line(size = 1, show.legend=FALSE) +
theme(axis.text.y=element_blank())
g3 <- ggplot(data, aes(x = x, y = fourier, color = function_label)) +
geom_line(size = 1, show.legend=FALSE) +
theme(axis.text.y=element_blank())
plot_grid(g1, g2, g3, ncol = 3)
What bases do you think will work best for \(f1\), \(f2\) and \(f3\)?
Answer: \(f1\) was made from polynomial bases, \(f2\) from fourier, \(f3\) from linear splines
