Deep Learning from Theory to Practice: Neural Networks, Logistic Regression, TensorFlow and Keras for Cat Image Classification

Abstract

With the rapid development of artificial intelligence over the past decade, machine‑learning frameworks such as TensorFlow, PyTorch, CNTK, and Keras have become increasingly developer‑friendly, making the deployment of deep learning to end‑devices feasible. This article introduces neural networks from basic concepts to practical programming, targeting beginners who have programming experience, basic linear‑algebra knowledge, and familiarity with TensorFlow/Keras and Jupyter Notebook.

Reading Tips

Any programming experience

Basic understanding of geometric algebra

Familiarity with TensorFlow/Keras (optional)

Familiarity with Jupyter Notebook (optional)

Content Outline

The material is divided into four chapters, each taking about 30 minutes to read:

Important concepts and mathematical derivations of neural networks

Practical implementation of a logistic‑regression cat detector

Shallow neural network with TensorFlow for cat detection

Deep neural network with Keras for cat detection

All practical code is executed in Jupyter Notebook.

Chapter 1 – Foundations

The perceptron, invented by Frank Rosenblatt in 1957, is the simplest feed‑forward neural network and a binary linear classifier. It learns a separating hyperplane by minimizing a loss function using gradient descent.

Biological Neuron

The perceptron abstracts a biological neuron, which consists of dendrites, synapses, soma, and axon. Input signals weighted by synapse strength are summed; if the total exceeds a threshold, the neuron fires, producing an action potential.

Artificial Neuron

In artificial neural networks, a single‑layer perceptron is a linear classifier; its main limitation is inability to solve non‑linearly separable problems such as XOR.

Logistic Regression

Logistic regression is a binary classification algorithm. The hypothesis function uses the sigmoid activation:

def sigmoid(z):
    s = 1 / (1 + np.exp(-z))
    return s

The loss (cost) function for a single example is the cross‑entropy, and the overall cost is the average over all examples.

Gradient Descent

Gradient descent iteratively updates parameters w and b to minimize the cost function, moving in the direction of the steepest descent until convergence.

def propagate(w, b, X, Y):
    m = X.shape[1]
    A = sigmoid(np.dot(w.T, X) + b)
    cost = -1/m * np.sum(Y*np.log(A) + (1-Y)*np.log(1-A))
    dw = 1/m * np.dot(X, (A-Y).T)
    db = 1/m * np.sum(A-Y)
    grads = {"dw": dw, "db": db}
    return grads, cost

The optimization loop runs for a specified number of iterations, recording the cost every 100 steps.

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost=False):
    costs = []
    for i in range(num_iterations):
        grads, cost = propagate(w, b, X, Y)
        w = w - learning_rate * grads["dw"]
        b = b - learning_rate * grads["db"]
        if i % 100 == 0:
            costs.append(cost)
            if print_cost:
                print("Cost after iteration %i: %f" % (i, cost))
    params = {"w": w, "b": b}
    return params, grads, costs

The final prediction function thresholds the sigmoid output at 0.5:

def predict(w, b, X):
    m = X.shape[1]
    Y_prediction = np.zeros((1, m))
    A = sigmoid(np.dot(w.T, X) + b)
    for i in range(A.shape[1]):
        Y_prediction[0, i] = 1 if A[0, i] > 0.5 else 0
    return Y_prediction

Chapter 2 – Logistic Regression Cat Detector

Code loads the cat/non‑cat dataset, reshapes images to vectors of size 12288 (64×64×3), normalizes pixel values, and trains the logistic‑regression model.

import numpy as np
import matplotlib.pyplot as plt
import h5py
from PIL import Image
from loadH5File import load_data
%matplotlib inline

train_set_x, train_set_y, test_set_x, test_set_y, classes = load_data()

num_px = train_set_x.shape[1]
m_train = train_set_x.shape[0]
m_test = test_set_x.shape[0]

train_x_flatten = train_set_x.reshape(m_train, -1).T
test_x_flatten = test_set_x.reshape(m_test, -1).T
train_set_x = train_x_flatten/255.
test_set_x = test_x_flatten/255.

The model is trained with 2000 iterations and a learning rate of 0.005, achieving ~99% training accuracy and ~70% test accuracy. Learning‑rate curves are plotted to illustrate the effect of different rates.

Chapter 3 – Shallow Neural Network with TensorFlow

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from loadH5File import load_data
from PIL import Image
%matplotlib inline

train_set_x, train_set_y, test_set_x, test_set_y, classes = load_data()
# flatten and normalize as before
...
X = tf.placeholder(tf.float32, [None, img_size])
Y = tf.placeholder(tf.float32, [None, 1])
W = tf.Variable(tf.random_normal([img_size, 1]))
b = tf.Variable(tf.random_normal([1]))
logits = tf.matmul(X, W) + b
Y_pred = tf.sigmoid(logits)
cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=Y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.002).minimize(cost)
...

Training runs for 6000 iterations with batch size 30, reporting training and test accuracies (≈64% test accuracy). Loss and accuracy curves are plotted.

Chapter 4 – Deep Neural Network with Keras

from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
import numpy as np
import matplotlib.pyplot as plt
from loadH5File import load_data
from PIL import Image
%matplotlib inline

train_set_x, train_set_y, test_set_x, test_set_y, classes = load_data()
# flatten and normalize as before
...
model = Sequential()
model.add(Dense(256, activation='relu', input_shape=(img_size,)))
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer=SGD(), loss='binary_crossentropy', metrics=['accuracy'])
history = model.fit(train_set_x, train_set_y, epochs=20, validation_data=(test_set_x, test_set_y))

The Keras model reaches up to ~83% training accuracy but test accuracy fluctuates, illustrating over‑fitting. Training/validation loss and accuracy curves are plotted.

Conclusion

The tutorial demonstrates that after a certain accuracy threshold, further improvements become difficult due to over‑fitting (high variance). Common remedies include L2 regularization, dropout, data augmentation, and early stopping. For beginners, high‑level frameworks like Keras are recommended for rapid prototyping, while TensorFlow offers more control for custom models.