Mastering Multiple Linear Regression: Theory, Estimation, and Prediction

Multiple Linear Regression Model

Multiple regression analysis is a statistical method for studying the relationship between random variables. By analyzing observed data, a quantitative relationship (regression equation) between one dependent variable and a set of independent variables is established, which can be used for prediction and control after statistical significance is confirmed.

Assuming a random variable Y is related to variables X₁,…,X_k, the multiple linear regression model can be expressed in matrix form Y = Xβ + ε, where ε follows a normal distribution and β_i are the regression coefficients.

The main steps of regression analysis are:

Determine the estimates of the parameters (regression coefficients) from the observed data;

Perform statistical tests on the linear relationship and the significance of the independent variables;

Use the regression equation for prediction.

Least Squares Estimation of Regression Coefficients

Given a sample of n observations, the model can be written as Y = Xβ + ε. The parameters are estimated by the least‑squares method, choosing β̂ that minimizes the sum of squared errors (SSE = (Y‑Xβ̂)ᵀ(Y‑Xβ̂)).

Setting the derivative to zero yields the normal equations XᵀXβ̂ = XᵀY. When the design matrix X has full column rank, XᵀX is invertible and the solution is β̂ = (XᵀX)⁻¹XᵀY.

The fitted values are Ŷ = Xβ̂, and the residuals are e = Y‑Ŷ. The residual sum of squares (RSS) is eᵀe.

Testing the Regression Equation and Coefficients

After fitting the model, it is necessary to test whether Y and the X variables have a linear relationship and whether each coefficient is significant. The total sum of squares is decomposed into regression sum of squares (SSR) and residual sum of squares (SSE).

At a chosen significance level α, the F‑statistic (F = (SSR/p) / (SSE/(n‑p‑1))) is compared with the critical value F_{α, p, n‑p‑1}; if F exceeds the critical value, the regression is considered significant.

If the overall null hypothesis is rejected, individual coefficients are tested using t‑statistics. Variables with non‑significant coefficients may be removed and the model refitted.

The coefficient of determination R² = SSR / SST measures the proportion of variance explained; values close to 1 (e.g., >0.9) indicate a strong linear relationship.

Prediction with the Regression Equation

For a given set of predictor values, substitute them into the regression equation to obtain the predicted response. Confidence intervals for the mean response and prediction intervals for a new observation can also be constructed.

References

司守奎，孙玺菁. Python数学实验与建模.