Mastering the Chain Rule for Vector‑to‑Vector and Scalar‑to‑Matrix Derivatives

Chain Rule for Vector‑to‑Vector Derivatives

When multiple vectors depend on each other, the derivative follows a chain rule where the Jacobian of the outer vector multiplied by the Jacobian of the inner vector yields the overall derivative. This rule extends to any number of dependent vectors, provided all variables are vectors; it does not hold if any variable is a matrix.

Scalar‑to‑Multiple‑Vector Chain Rule

In machine‑learning loss functions, the final target is a scalar. Directly applying the vector‑to‑vector chain rule can lead to dimension mismatches. By transposing the scalar derivative term, dimensions become compatible, resulting in a chain rule that expresses the scalar derivative as a product of transposed Jacobians and vectors. This formulation works for any number of vector arguments.

Scalar‑to‑Multiple‑Matrix Chain Rule

Deriving a scalar with respect to several matrices is more complex because matrix‑to‑matrix derivatives are not as straightforward. Instead, one can apply the scalar‑to‑vector chain rule to each element of the matrices or use definition‑based methods. The resulting expressions involve indicator functions that are 1 when indices match and 0 otherwise, ultimately yielding inner‑product forms between rows and columns of the involved matrices.

Matrix‑Vector Derivative Summary

Three primary methods exist for matrix‑vector differentiation: definition‑based, differential‑based, and chain‑rule‑based. When possible, the chain‑rule approach—especially the four key conclusions presented—is preferred for its efficiency. If no suitable chain rule applies, the differential method is the next choice, and the definition method serves as a fallback.