Unlocking Matrix Differential Calculus: A Practical Guide to Trace‑Based Derivatives

Matrix Differential

In calculus we first learn scalar derivatives and differentials, where the differential of a scalar function f can be written as df = f' dx. For multivariable functions the differential can be expressed as a sum of partial derivatives multiplied by differential increments.

Extending this idea, the derivative of a scalar with respect to a vector and its vector differential are related by a transpose. We now generalize to matrices. The definition of a matrix differential is

\[ d\mathbf{Y} = \mathrm{tr}\big( (\partial \mathbf{Y} / \partial \mathbf{X})^T d\mathbf{X} \big) \]

where the second step uses the property of the trace function, i.e., the trace equals the sum of the diagonal elements.

From the matrix differential formula we see that matrix differentials and their derivatives also have a transpose relationship, wrapped by a trace function. Since the trace of a scalar is the scalar itself, matrix differentials and vector differentials can be expressed uniformly as

Properties of Matrix Differentials

Before applying matrix differentials to compute derivatives, we list their main properties:

Differential addition and subtraction

Differential multiplication

Differential transpose

Differential of a trace

Differential of a Hadamard product

Element‑wise differentiation

Differential of an inverse matrix

Differential of a determinant

Using the Differential Method for Matrix‑Vector Derivatives

Given a scalar function that is composed of matrix operations (addition, multiplication, inverse, determinant, element‑wise functions, etc.), we first compute its differential using the appropriate rules, then apply the trace trick: wrap the differential with a trace and move remaining terms to the left side. For the part of the trace that appears on the left, adding a transpose yields the desired derivative.

The trace tricks we use are:

The trace of a scalar equals the scalar itself.

The trace is invariant under transposition.

Cyclic property of the trace: \(\mathrm{tr}(AB)=\mathrm{tr}(BA)\) when dimensions match.

Linearity of the trace (addition/subtraction).

Example workflow:

Apply the differential multiplication property to obtain the differential of the target expression.

Wrap both sides with the trace function.

Use trace properties 1 and 3 to rearrange terms.

Identify the left‑hand part of the trace; adding a transpose gives the derivative.

This process avoids differentiating each individual scalar element of the matrix, making the computation more convenient.

Summary of the Differential Method

Matrix differentials allow us to compute derivatives without element‑wise differentiation, provided we are familiar with the listed properties and can skillfully apply trace tricks. For more complex, multi‑layer chain‑rule scenarios, combining known simple derivative results with the chain rule can further simplify the work, which will be covered in the next article.