Unlocking Multiple Linear Regression: Theory, Estimation, and Prediction

Multiple Linear Regression Model

Multiple regression analysis is a statistical method for studying the relationships among random variables. By analyzing observed data, a quantitative relationship (regression equation) between one dependent variable and a set of independent variables is established; after statistical testing confirms significance, the model can be used for prediction and control.

Assuming the random variable Y is related to variables X₁, X₂, …, Xₚ, the multiple linear regression model is expressed as Y = β₀ + β₁X₁ + … + βₚXₚ + ε, where ε follows a normal distribution and β₀,…,βₚ are the regression coefficients. The main steps of regression analysis are:

Determine the estimates of the parameters (regression coefficients) from the observed data.

Perform statistical tests on the linear relationship and the significance of each independent variable.

Use the regression equation for prediction.

Least‑Squares Estimation of Regression Coefficients

Given a sample of n observations of the dependent variable Y and the p independent variables X₁,…,Xₚ, we form the data matrices and substitute them into the model. The error terms ε₁,…,εₙ are independent and identically normally distributed.

Let X denote the design matrix and y the vector of observed responses. The model can be written in matrix form as y = Xβ + ε, where Iₚ is the p‑dimensional identity matrix.

The parameters are estimated by the ordinary least‑squares method, selecting β̂ that minimizes the sum of squared errors S(β) = (y – Xβ)ᵀ(y – Xβ). Setting the derivative to zero yields the normal equations XᵀXβ̂ = Xᵀy. When XᵀX is of full rank, it is invertible and the solution is β̂ = (XᵀX)⁻¹Xᵀy.

The fitted values are ŷ = Xβ̂, and the residuals e = y – ŷ. The residual sum of squares (RSS) is eᵀe.

Testing the Regression Equation and Coefficients

After fitting the model, we must test whether the assumed linear relationship holds and whether each independent variable significantly contributes. Decomposing the total sum of squares gives SST = SSR + SSE, where SSE is the residual sum of squares (reflecting random error) and SSR is the regression sum of squares (reflecting the effect of the independent variables).

To test the overall significance, we use the F‑statistic: F = (SSR / p) / (SSE / (n – p – 1)). At a chosen significance level α, if F exceeds the critical value from the F‑distribution, the regression model is considered significant.

If the overall null hypothesis is rejected, we may further test individual coefficients using t‑statistics or perform stepwise procedures to identify non‑significant variables, removing them and refitting the model. The coefficient of determination R² = SSR / SST measures the proportion of variance explained; values close to 1 (e.g., >0.9) indicate a strong linear relationship.

Prediction with the Regression Equation

For a given set of predictor values x₀, the predicted response is ŷ₀ = β̂₀ + β̂₁x₀₁ + … + β̂ₚx₀ₚ. Prediction intervals can also be constructed. Let σ̂² be the estimated error variance; then the (1–α) confidence interval for the mean response at x₀ is ŷ₀ ± t_{α/2, n‑p‑1}·σ̂·√(1 + x₀ᵀ(XᵀX)⁻¹x₀). When the sample size is large, the interval approximates a 95% prediction interval.

References

司守奎，孙玺菁. Python数学实验与建模.