Demystifying Neural Networks: A Mathematical Approach (Part 1)

In this article we discuss the mathematics behind simple neural networks, illustrating how mathematics plays a crucial role when building our own AI models.

Before diving into neural networks, we first introduce two elementary building blocks – a simple predictor and a basic classifier – to lay the groundwork for later concepts.

Simplified Predictor

The problem statement is to convert kilometres to miles using a device that implements a linear relationship. The core equation is miles = kilometres * c , where c is an unknown constant that we need to determine.

By constructing a truth table (training data) of kilometre‑mile pairs, we aim to compute the constant c . Several candidate values (0.5, 0.6, 0.7) are tested, and the resulting errors are compared. The analysis shows that a value of c = 0.6 yields the smallest error, while larger values lead to over‑adjustment.

Simplified Classifier

We then consider a classification task: distinguishing two types of garden bugs (caterpillars vs. ladybugs) based on their length and width. The goal is to find a line that separates the two classes.

Starting with a random line y = 0.25x , we evaluate its performance on the training data and compute the error. By adjusting the slope (gradient) of the line – denoted A – we iteratively reduce the error. For example, using the point (3.0, 1.1) as a target, the error is E = 1.1 – 0.75 = 0.35 , leading to an updated slope A = 0.3667 .

Simplified Multi‑Classifier

The article explains the limitation of a single linear classifier for non‑linearly separable problems such as the XOR Boolean function. While AND and OR can be represented by a straight decision boundary, XOR requires at least two linear pieces, motivating the need for multiple classifiers working together – the essence of a neural network.

Simple Neural Network

Neurons are presented as the fundamental building blocks of neural networks. Each neuron receives multiple inputs, computes a weighted sum, and passes the result through an activation function. Several activation functions are described:

Step function – outputs zero below a threshold and jumps to one above it.

Sigmoid – y = 1/(1+e^-x) , smooth but suffers from vanishing gradients at the extremes.

Tanh – tanh(x)=2/(1+e^(-2x)) -1 , symmetric around zero, also prone to vanishing gradients.

ReLU – f(x)=max(0,x) , widely used due to its computational efficiency and sparse activation, though it can encounter dying‑neuron issues.

The article chooses the sigmoid function for simplicity in the subsequent examples.

Modeling an Artificial Neuron

A typical artificial neuron computes output = activation( Σ w_i * x_i + b ) , where w_i are the connection weights and b is a bias term. By arranging neurons in multiple layers (e.g., a three‑layer network with three nodes per layer), the network can represent complex mappings. Training adjusts the weights – increasing them to amplify signals or decreasing them to suppress noise – until the network’s output matches the desired targets.

Conclusion

The article demonstrates how simple predictors and classifiers form the foundation for understanding neural networks. It also highlights the biological inspiration behind artificial neurons, the importance of activation functions, and the role of weight adjustment during learning.