3.3.1 RSS: Residual Sum Squared

Sum of Squared residuals

\[RSS(\hat{\beta}) = e^t e \]

\[e^Te = (Y-X\hat{\beta})^T(Y-X\hat{\beta})\]

Set derivative to 0 to find inflection point

\[\frac{\partial e^Te}{\partial\beta}=-2X^TY+2X^TX\hat{\beta} = 0\]

\[(X^TX)\hat{\beta}=X^T Y\]

\[ \hat{Y} = X\hat{\beta}=(X^T X)^{-1} X^T Y\]

\[\hat{Y} = (X^T X)^{-1} X^T ( X \beta + \epsilon )\]

Show that this inflection point is minimum by proving the 2nd derivative is positive (or positive definite for matrices).

\[\frac{\partial^2 e^Te}{\partial\beta \partial\beta^T} = 2X^T X\]
Assuming X has full column rank, \(X^T X\) can be shown to be positive definite.

3.3.2 Covariance matrix

The \(X^T X\) is the Covariance Matrix in multivariate normal distribution.

$$= Cov(X,X) = (X_i - _i)(X_j-_j)

\[ E[Y-\hat{Y}]^2 = E[f(X)+\epsilon -\hat{f}(X)]^2 = {\color{red}E[f(X)-\hat{f}(X)]^2} + Var(\epsilon)\]

\[{\color{red}e = f(X)-\hat{f}(X)}\]

• reducible error : \({\color{red}E[f(X)-\hat{f}(X)]^2}\)
• irreducible error: \(Var(\epsilon)\)

Steps:

1. Choose model: Linear equation

\[ Y = f(X_1,X_2,...) + \epsilon\]

\(f\) is matrix multiplication with weight vector \(\beta\)

\[Y = X\beta + \epsilon\]

2. Train parameters: Ordinary Least Squares(OLS)

a <- c(2,3,5,6)

library(ISLR)



x <- rnorm(50,mean = 0,sd = 1)  # creates 50 points
y <- x + rnorm(50,mean=1,sd=0.5) # creates 50 points 
cor(x,y) # correlation
#> 0.8344

3.4.1 T-test

• T-dist is a probability dist like a normal-dist but bigger tails
  • infinite degree of freedom result in T-dist = normal-dist
• smaller degree of freedoms imply bigger tails
• Can be useful to model returns with fat fails

Null H: There is no linear relationship between independent variable and output \(\beta = 0\)
Alt H: There is a linear relationship between independent variable and output \(\beta \neq 0\)

We do a T-test for EACH coefficient.
What happens when only some coefficient are significant and others arent.
ANSWER: Typically, we throw out nonsignificant (high p-value) coefficients to sacrifice some bias to reduce variance.
Remember high variance leads to overfit.

3.4.2 Validate multi-linear regression model

3.5.1 MSE

• MSE(Mean Squared Error) : used to check how well our regression model fits
  • Add the squared distance from point to regression, then divide by count of datapoints
• Notice \(MSE = \frac{1}{n} RSS\)

Goal is to minimize MSE of test dataset, not training dataset

3.5.2 R^2

\(R^2\) is a number in [0,1] and related to Variance(Scatter or Clumped)
\(R^2\) measure how scattered or clumped the data WRT our regression line aka model.

• Close to 1 means every datapoint is on our regression line
• Close to 0 means the datapoint are scattered but broadly follow trend with our regression line.

Optional: Drop coefficients aka Independent Random variables that do not contribute well in \(R^2\) despite having low p-value

Example: Age variable gives \(R^2\) 0.7 with p-value: 0.04 Adding a height variable gives \(R^2\) 0.71 with p-value 0.005
We can safely drop height variable

Cons:

• R^2 increases as you add more parameters
  • Solution: used Adj-R^2

3.5.3 Cor

\[R^2 = Cor(Y,\hat{Y})^2 \text{ for multiple lin reg}\]
Linear fit models aim to \(max(Cor(Y,\hat{Y}))\)