Understanding Computational Graphs and Automatic Differentiation for Neural Networks

Neural network structures are not limited to fully‑connected layers; modern deep learning also uses local connections, weight sharing, and skip connections. To train any neural network, we first need a flexible description and a method to compute gradients.

A computational graph is a directed acyclic graph where nodes represent variables and edges represent operations. By connecting a network and its loss function into a graph, automatic differentiation can compute the Jacobian of the loss with respect to each parameter.

The article introduces computational graphs with examples: a fully‑connected network, a simple convolutional network, and a non‑fully‑connected network. It shows how each node can have at most two parents after transformation, and how vector and matrix nodes can express complex computations.

Automatic differentiation is performed by traversing the graph backward. Each node stores its value and the Jacobian of the final result with respect to itself. The backward pass accumulates Jacobians using the chain rule, caching intermediate results to avoid redundant computation. Pseudocode for forward and backward propagation is provided.

Implementation details are given in pure Python with NumPy. A base Node class defines parent/child relationships, forward evaluation, and abstract methods for computing values and Jacobians. Subclasses such as Variable , Add , Dot , MatMul , Logistic , ReLU , SoftMax , and CrossEntropyWithSoftMax implement specific operations and their Jacobians.

A simple Graph class stores all nodes and provides utilities to clear Jacobians, reset values, and visualize the graph. Optimizer classes inherit from a base Optimizer and implement gradient accumulation over mini‑batches. Concrete optimizers include GradientDescent , RMSProp , and Adam , each updating trainable Variable nodes using the accumulated gradients.

Finally, the article builds a multilayer fully‑connected neural network for MNIST digit classification using the custom graph framework. The network consists of two hidden ReLU layers and a soft‑max output layer, trained with the Adam optimizer. Training and evaluation loops are shown, and the resulting computational graph is visualized.

The conclusion emphasizes that any neural network can be expressed as a computational graph, and automatic differentiation (the generalized back‑propagation) provides an efficient way to compute gradients for training.