Collinearity and Inference
Why is multicollinearity bad in standard linear regression? Explain by describing how it affects the following quantities: (1) the variance of the fitted regression coefficients, (2) $t$-statistics for individual coefficients.
Answer
Multicollinearity means the predictors are nearly linearly dependent, so $X^\top X$ is close to singular. This makes coefficient estimates unstable and inference unreliable, even if the overall fit looks good.
Variance of the coefficients: increases.
\[ \widehat{\beta}=(X^\top X)^{-1}X^\top y, \qquad \mathrm{Var}(\widehat{\beta}\mid X)=\sigma^2 (X^\top X)^{-1}. \]When predictors are multi-colinear, $X^\top X$ becomes ill-conditioned (nearly singular), so $(X^\top X)^{-1}$ has very large entries (due to determinant being close to zero). Hence the variance of the coefficients increases.
$t$-statistics for individual coefficients: typically decrease in magnitude.
\[ t_j=\frac{\widehat{\beta}_j}{\mathrm{SE}(\widehat{\beta}_j)}. \]Because multicollinearity increases $\mathrm{SE}(\widehat{\beta}_j)$, the denominator becomes large, so $|t_j|$ tends to shrink (this of course depends on the sample at hand).
As a result, individual predictors may appear insignificant (the $p$-value becomes larger, so we fail to reject the null hypothesis more often, implying less statistical significance for the individual coefficients) even when they matter jointly (which can happen). In such cases, joint significance can be tested using an $F$-test.