Gradient Descent

In one sentence: Gradient descent nudges a model’s settings a small step in the direction that lowers the error, then repeats until the error stops falling.

$\theta_{\text{new}} = \theta_{\text{old}} - \eta \, \nabla L(\theta_{\text{old}})$

Read it out loud: “Theta new equals theta old, minus eta times the gradient of L, evaluated at theta old.”

What each symbol means

Symbol	Meaning
$\theta$	The settings the model learns, such as its weights and biases. When there are many settings, they are gathered into one list. Training means finding good values for them.
$\theta_{\text{old}}$	The current settings, before this update step.
$\theta_{\text{new}}$	The settings after one update step.
$\eta$	The learning rate, the Greek letter eta: a small positive number that sets how big each step is.
$L$	The loss function, which scores how wrong the model is at a given set of settings. Lower is better.
$\nabla L$	The gradient of the loss: the slope of the loss in each direction. It points the way the loss rises fastest.

What the formula is doing

Training a model means finding settings $\theta$ that make the loss $L$ small. The loss is a score built from the model’s errors, and it depends on the settings. Picture the loss as a landscape of hills and valleys, where the height is the loss and each position is one choice of settings.

The gradient $\nabla L$ tells you the slope at your current position. It points uphill, in the direction the loss rises fastest. To lower the loss, the update goes the opposite way, and that is the minus sign in the formula.

The learning rate $\eta$ controls the size of each step. A small $\eta$ takes safe, tiny steps that can be slow. A large $\eta$ takes big steps that can jump over the valley and make the loss worse. A good value is usually found by trying a few.

The formula is applied again and again. Each step moves the settings a little downhill. As the slope flattens near the bottom, the steps shrink and the loss stops falling much. That is the sign that training has reached a low point.

An everyday picture

Think of walking down a foggy hill in the dark. You cannot see the bottom, but you can feel which way the ground slopes under your feet. You take a step downhill, feel the slope again, and repeat. Long strides get you down quickly but risk stumbling past the bottom; short steps are safe but slow.

Worked example: Finding the bottom of a simple bowl

The loss is $L(w) = (w - 3)^2$ , a bowl whose lowest point is at $w = 3$ . Its gradient is $\frac{dL}{dw} = 2(w - 3)$ . Start at $w = 0$ with learning rate $\eta = 0.1$ . All values are unitless here.

First step: the gradient at $w = 0$ is $2(0 - 3) = -6$ . The update is $w = 0 - 0.1 \times (-6) = 0.6$ .
Second step: the gradient at $w = 0.6$ is $2(0.6 - 3) = -4.8$ . The update is $w = 0.6 - 0.1 \times (-4.8) = 1.08$ .
Third step: the gradient at $w = 1.08$ is $2(1.08 - 3) = -3.84$ . The update is $w = 1.08 - 0.1 \times (-3.84) = 1.464$ .

Answer: After three steps the setting is $w = 1.464$ , closer to the bottom at 3 than the start was. Each step is smaller than the one before, because the slope gets gentler as $w$ nears 3.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Adding the gradient instead of subtracting it. That climbs the hill, so the loss grows.
Using a learning rate that is too big. In the bowl above, with $\eta = 1.1$ , the first step jumps from $w = 0$ to $w = 6.6$ , and the loss rises from 9 to 12.96. The next step lands at $w = -1.32$ , and the steps keep growing, so training blows up instead of settling.
Using a learning rate that is too small and not noticing. Training crawls, looks stuck, and wastes time. Watch how the loss changes over the steps.
Confusing the loss with the gradient. The loss is one number for a given setting. The gradient is the slope, and it changes as the setting changes.

Where you will meet it

Training the weights of nearly every neural network, usually in a variant called stochastic gradient descent.
Fitting linear and logistic regression models.
Large language and image models, which run huge numbers of these updates during training.
Curve fitting in science and engineering, where the parameters of a model are tuned to match measured data.

Questions people ask

How do you know the learning rate is right?

Watch the loss. If it falls steadily, the rate is fine. If it bounces up and down or grows, the rate is too big. If it falls very slowly, the rate is probably too small.

Does gradient descent always find the best answer?

No. On a bumpy landscape it can settle into a low point that is not the lowest one. On a simple bowl like the example, it reaches the bottom if the steps are small enough.

Where does the gradient come from?

From calculus. It is the set of partial derivatives of the loss with respect to each setting. Backpropagation is the method neural networks use to compute them efficiently.

What is stochastic gradient descent?

It is the same update, but the gradient is estimated from a small random batch of data rather than the whole dataset. Each step is cheaper, and the noise in the estimate often helps.

Mean Squared Error (MSE): Mean squared error measures how wrong a model’s guesses are by squaring each miss, so big misses count much more, and then averaging them.
Cross-Entropy Loss: Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.
Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds them up with a bias, and then decides how strongly to fire.