Unlocking Nonlinear Insights: A Practical Guide to Generalized Additive Models (GAM)

Generalized Additive Model

Model Definition

Classical linear regression assumes a linear relationship between features and the response variable, which often does not hold in practice. For example, calcium supplementation improves health up to a point, after which excess calcium becomes harmful, producing a non‑linear, inverted‑U relationship. To handle such cases, Stone (1985) introduced additive models and Hastie (1990) proposed Generalized Additive Models (GAM). The relationship between additive models and GAM mirrors that of linear regression and generalized linear models: when the link function is the identity, a GAM reduces to an additive model, making additive models a special case of GAMs.

Before fitting a model, one can plot a scatter diagram of a feature versus the response to inspect their relationship. When a nonlinear relationship exists but its exact functional form is unknown, GAM provides a flexible alternative by incorporating non‑parametric smooth terms.

In a GAM, the model can be written as:

g(E[Y]) = β0 + f1(X1) + f2(X2) + ... + fp(Xp)

where β0 is the intercept, each fi(·) is a smooth, non‑parametric function (often represented by splines, kernels, or local regression), and g(·) is a link function that must be smooth and invertible. For instance, if the response follows a Poisson distribution, a log link is commonly used.

The flexibility of GAMs comes from the link function, but fitting a smooth term for every feature in high‑dimensional data can be computationally expensive and prone to over‑fitting. Therefore, semi‑parametric GAMs combine linear terms for some features with smooth terms for others:

g(E[Y]) = β0 + Σ_{j∈L} βj Xj + Σ_{k∈N} fk(Xk)

where indices in L correspond to linear features and indices in N correspond to nonlinear features, allowing the model to capture both linear and nonlinear effects.

Model Estimation

When the link function is the identity, a GAM collapses to an additive model. Estimation of additive models relies on the backfitting algorithm, which iteratively updates each smooth component while holding the others fixed.

The backfitting procedure can be summarized as:

Initialization: set all smooth functions to zero
Iterate: for each component j = 1,…,p do
  compute residuals r = Y – Σ_{k≠j} fk(Xk)
  update fj by smoothing (Xj, r)
Until convergence criteria are met

GAMs extend this approach by using a local scoring algorithm: an outer loop estimates the link function via Fisher scoring, while an inner loop applies backfitting to estimate the smooth functions.

Model Interpretability

GAMs offer strong interpretability because each feature’s effect is represented by its own smooth function. For example, using two features X1 and X2, one can plot the estimated smooth curves to assess whether the relationships are linear or nonlinear.

A semi‑parametric model might yield results such as: (1) a unit increase in X1 raises the response by a fixed amount; (2) the effect of X2 is captured by a smooth function that cannot be expressed in a simple closed form, showing a pattern of slight decrease, increase, then decrease as X2 grows.

These insights allow practitioners to understand how different feature values influence the response and to provide business‑oriented interpretations.

Advantages and Limitations

Advantages of GAMs include: (1) they do not require a predefined parametric relationship between features and response, making them flexible for nonlinear data; (2) the additive decomposition yields high interpretability.

Limitations include: (1) the use of non‑parametric smooth functions increases computational cost; (2) compared with some modern machine‑learning models, GAMs may achieve lower predictive accuracy.

Reference:

Shaoping, Yang Jianyong, Su Sida. Interpretable Machine Learning: Models, Methods, and Practice .