Understanding Gradient Descent: Concept and Implementation

We assume a hypothesis function and a way to measure its fit to data; estimating the parameters of this function leads us to use gradient descent.

Concept

Imagine plotting the hypothesis parameters θ0 and θ1 as axes, with the cost function represented on the vertical axis; each point on the surface corresponds to the cost for a particular pair of parameters.

When the cost surface reaches its lowest point (the bottom of a “valley”), the algorithm has converged, as indicated by the red arrow in the illustration.

Gradient descent follows the negative gradient of the cost function: the derivative at a point gives the direction of steepest descent, and the step size is controlled by the learning rate α.

Small α yields small steps, large α yields larger steps; the algorithm repeatedly updates the parameters in the direction of the gradient until it reaches a local minimum.

Visualize the cost surface as a mountain; at each position you look around, decide the direction of the steepest descent, take a small step, and repeat until you arrive at the lowest point.

The learning rate α determines how large each step is; a large α may overshoot, while a small α results in slow convergence.

Gradient descent algorithm: repeat until convergence.

During each iteration, update each parameter θi sequentially after computing its gradient; updating parameters in the wrong order can lead to incorrect implementations.

Always compute all gradients first, then update the parameters.