Backpropagation and the Chain Rule

In one sentence: Backpropagation finds how much each weight affects the error by multiplying the small slopes along the chain from that weight to the loss, one link at a time.

$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat y} \, \frac{\partial \hat y}{\partial z} \, \frac{\partial z}{\partial w}$

Read it out loud: “The partial derivative of L with respect to w equals the partial derivative of L with respect to y-hat, times the partial derivative of y-hat with respect to z, times the partial derivative of z with respect to w. Here L is the loss, the error score, and w is a weight. Y-hat is the prediction, and z is the weighted sum before the sigmoid. A partial derivative is a slope measured while every other setting is held still.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Neuron, weight, bias and weighted sum $z$: A neuron is a small calculation unit in a neural network. It takes an input $x$ and multiplies it by a weight $w$ , which says how strongly the input counts. It adds a bias $b$ , an extra number that shifts the result. The weighted sum is $z = wx + b$ . For example, with $w = 0.5$ , $x = 2$ and $b = -0.5$ , we get $z = 0.5 \times 2 - 0.5 = 0.5$ .
The sigmoid $\sigma$ and the number $e$: The sigmoid, written $\sigma$ (Greek sigma), turns any number into a value between 0 and 1: $\sigma(z) = 1 / (1 + e^{-z})$ . Here $e$ is Euler’s number, about 2.718, and $e^{-z}$ means $1$ divided by $e^{z}$ . A large positive $z$ gives a value close to 1, and a large negative $z$ gives a value close to 0. Its graph is S-shaped.
Partial derivatives and slopes, like $\frac{\partial L}{\partial w}$: A derivative measures how fast a quantity changes when its input changes by a tiny amount. It is a slope. The curly symbol $\partial$ is read partial, and $\frac{\partial L}{\partial w}$ is read partial L over partial w. It means: change only $w$ by a tiny amount, keep every other setting still, and measure how much $L$ changes. A large value means the loss is very sensitive to $w$ .
The chain rule: The chain rule from calculus says that when a quantity depends on a second one, which depends on a third one, the slopes multiply along the chain. For example, if $z$ changes twice as fast as $w$ , and $y$ changes three times as fast as $z$ , then $y$ changes six times as fast as $w$ . Backpropagation uses this rule for every weight in a network.
Loss $L$ and squared error: The loss $L$ is one number that scores how wrong a prediction is, and lower is better. Squared error is one common loss: $L = (\hat{y} - y)^{2}$ , the square of the gap between the prediction and the correct answer. Squaring makes the loss zero when the prediction is exact and grows quickly as the gap gets bigger.
Target $y$ and prediction $\hat{y}$: The target $y$ is the correct answer the neuron should give. The prediction, written $\hat{y}$ and read y-hat, is the neuron’s actual output. The hat marks a prediction, not a true value. The gap between them, $\hat{y} - y$ , is the miss. In the example the target is 1, so the neuron should produce an output as close to 1 as it can.
Gradient descent and the learning rate $\eta$: The gradient is a slope, and gradient descent is the rule that moves each weight a small step opposite to its slope, so the loss goes down. The Greek letter eta, $\eta$ , is the learning rate: a small positive number that sets how big each step is. Backpropagation supplies the slopes, and gradient descent uses them to change the weights.

What each symbol means

Symbol	Meaning
$L$	The loss, a single number that scores how wrong the prediction is. Here it is squared error, $L = (\hat{y} - y)^{2}$ . It has no unit in this toy example, but in real problems it has the unit of the target squared. Lower is better.
$\frac{\partial L}{\partial \hat{y}}$	How much the loss changes when the prediction changes by a tiny amount, with everything else held fixed. For squared error it equals $2(\hat{y} - y)$ . A negative value means that raising the prediction would lower the loss.
$\frac{\partial L}{\partial w}$	The slope of the loss with respect to the weight $w$ , measured while all other settings stay fixed. It says how much the loss changes when $w$ changes by a tiny amount. Gradient descent moves $w$ in the opposite direction to this slope.
$\hat{y}$	The prediction, the neuron’s output after the sigmoid: $\hat{y} = \sigma(z)$ . It is always between 0 and 1, because the sigmoid is. It can be read as the neuron’s confidence that the answer is 1.
$\frac{\partial \hat{y}}{\partial z}$	How much the prediction changes when the weighted sum $z$ changes by a tiny amount. For the sigmoid it equals $\sigma(z)(1 - \sigma(z))$ . It is largest, 0.25, at $z = 0$ , and close to 0 far from zero.
$z$	The weighted sum before the sigmoid, $z = wx + b$ . It is a pure number in this toy example and can be any number, positive or negative. It is also called the pre-activation.
$\frac{\partial z}{\partial w}$	How much the weighted sum changes when the weight $w$ changes by a tiny amount. Because $z = wx + b$ , this equals the input $x$ . A bigger input means the weight has a bigger effect on the weighted sum.
$w$	A weight, one of the settings the neuron learns. It scales the input $x$ . In the example $w = 0.5$ , and training changes it step by step to reduce the loss.
$x$	The input given to the neuron, a number such as 2. It is the value that the weight scales, and it is also the slope of the weighted sum with respect to the weight.
$b$	The bias, a second setting that is added to the weighted input. In the example $b = -0.5$ . Its slope uses the same chain, but its last link is 1 instead of $x$ , because the bias adds straight to $z$ .
$y$	The target, the correct answer the neuron should output. In the example $y = 1$ , so training pushes the prediction toward 1.
$\sigma$	The sigmoid function, $\sigma(z) = 1 / (1 + e^{-z})$ . It takes any number and returns a value between 0 and 1 along an S-shaped curve. It has no unit.

The formula piece by piece

$\frac{\partial z}{\partial w}$: The smallest link: how the weighted sum moves when the weight moves. Because $z = wx + b$ , this link is the input $x$ itself. In the example it equals 2, so a change of 1 in $w$ changes $z$ by 2.
$\frac{\partial \hat{y}}{\partial z}$: The next link: how the prediction moves when the weighted sum moves. For the sigmoid it equals $\hat{y}(1 - \hat{y})$ , which is about 0.2350 in the example, a fairly flat part of the S-curve.
$\frac{\partial L}{\partial \hat{y}}$: The link from the loss to the prediction. For squared error it equals $2(\hat{y} - y)$ . In the example it is about $-0.7551$ , which says that raising the prediction would lower the loss.
$\frac{\partial L}{\partial \hat{y}} \, \frac{\partial \hat{y}}{\partial z}$: Two links multiplied: how the loss changes with the weighted sum. The chain runs backward, from the loss to the prediction and then to the weighted sum, and the two slopes multiply.
$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat y} \, \frac{\partial \hat y}{\partial z} \, \frac{\partial z}{\partial w}$: The whole rule in one sentence: the slope of the loss with respect to the weight equals the slope of the loss with respect to the prediction, times the slope of the prediction with respect to the weighted sum, times the slope of the weighted sum with respect to the weight.

What the formula is doing, and why

What goes in and what comes out. A single neuron takes an input $x$ , forms $z = wx + b$ , and passes $z$ through a sigmoid to get a prediction $\hat{y}$ . The loss $L$ depends on $\hat{y}$ , $\hat{y}$ depends on $z$ , and $z$ depends on $w$ . Training needs to know how much the loss changes when $w$ changes. Backpropagation answers that by following the chain of dependencies backward, from the loss toward the weight, and multiplying the links along the way.

Why multiply? The chain rule says that when one quantity depends on a second, which depends on a third, the slopes multiply. The first link, $\frac{\partial L}{\partial \hat{y}}$ , says how fast the loss changes with the prediction. The second, $\frac{\partial \hat{y}}{\partial z}$ , says how fast the prediction changes with the weighted sum. The third, $\frac{\partial z}{\partial w} = x$ , says how fast the weighted sum changes with the weight. A small change in $w$ passes through each link, and each link scales it.

The three links for this neuron. For squared error $L = (\hat{y} - y)^{2}$ and a sigmoid output, the three links are $2(\hat{y} - y)$ , $\sigma(z)(1 - \sigma(z))$ and $x$ . Multiplying them gives the slope for $w$ . The bias $b$ uses the same chain, except that its last link is $\frac{\partial z}{\partial b} = 1$ instead of $x$ , because the bias is added to $z$ with no scaling.

Why this is efficient, and when it applies. A network with many layers repeats the same idea. The slope at the output passes back one layer at a time, and each layer multiplies it by its own local slopes and reuses the result from the layer after it. That reuse keeps the work manageable. The sigmoid example is one neuron, but the same chain rule works for any network whose steps have slopes. The slopes are then fed to gradient descent, which updates the weights.

An everyday picture

Think of a row of gears. Turn the first gear and the second turns by its gear ratio, and the third turns by its own ratio. To find how far the last gear moves when you turn the first, multiply the ratios along the chain. Backpropagation does the same with slopes: it multiplies the small effect of each step to find the effect of one weight on the final loss.

Worked example: One sigmoid neuron with squared error

A single neuron has input $x = 2$ , weight $w = 0.5$ and bias $b = -0.5$ . The target is $y = 1$ . The neuron computes $z = wx + b$ , then $\hat{y} = \sigma(z)$ , and its loss is $L = (\hat{y} - y)^{2}$ . We use a learning rate of $\eta = 0.1$ . All values are unitless, so think of them as pure numbers in a toy example.

Forward pass: compute the weighted sum. Data flows forward through the neuron first. We multiply the input by the weight, $0.5 \times 2 = 1$ , and then add the bias, $1 + (-0.5) = 0.5$ . So $z = 0.5$ , which is the number the sigmoid reads.
Forward pass: compute the prediction and the loss. The sigmoid of 0.5 is $\sigma(0.5) = 1 / (1 + e^{-0.5}) \approx 0.6225$ , a number between 0 and 1. The loss is $L = (0.6225 - 1)^{2} \approx 0.1425$ , the square of the gap to the target 1. The neuron is somewhat wrong, so training will try to fix it.
Link 1: how the loss depends on the prediction. For squared error the slope is $\frac{\partial L}{\partial \hat{y}} = 2(\hat{y} - y) = 2(0.6225 - 1) \approx -0.7551$ . The negative sign says that raising the prediction would lower the loss, which fits a prediction that is too small.
Link 2: how the prediction depends on the weighted sum. The sigmoid’s slope is $\hat{y}(1 - \hat{y}) = 0.6225 \times 0.3775 \approx 0.2350$ . This is fairly small because the sigmoid is already flattening at $z = 0.5$ . The slope is largest, 0.25, at $z = 0$ .
Link 3: how the weighted sum depends on the weight. Because $z = wx + b$ , the slope is $\frac{\partial z}{\partial w} = x = 2$ . Each unit of weight adds 2 units to the weighted sum, since the input is 2.
Multiply the links and take a step. The product is $\frac{\partial L}{\partial w} = (-0.7551) \times 0.2350 \times 2 \approx -0.3549$ . Gradient descent moves the weight the opposite way, so $w_{\text{new}} = 0.5 - 0.1 \times (-0.3549) \approx 0.5355$ . The loss with the new weight is about 0.1303, which is lower than 0.1425.

Answer: The gradient is negative, so increasing $w$ lowers the loss. The update raises the weight from 0.5 to about 0.5355, and the loss falls from about 0.1425 to 0.1303. Raising $w$ raises $z$ , which moves $\hat{y}$ closer to the target of 1.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Leaving out a link. The three slopes form a chain, so all of them must be multiplied. For the weight, the last link is $x$ , not 1.
Using the wrong sigmoid slope. The slope of $\sigma$ is $\sigma(z)(1 - \sigma(z))$ , not $\sigma(z)$ alone.
Using the weight’s chain for the bias without changing the last link. For the bias, $\frac{\partial z}{\partial b} = 1$ , because the bias adds directly to $z$ .
Getting the sign wrong. The gradient here is negative, and the minus sign in gradient descent turns that into an increase of $w$ . Forgetting that minus sign would move the weight the wrong way.

Where you will meet it

Training every layer of a neural network, from a single neuron to models with billions of weights.
Automatic differentiation in deep learning libraries such as PyTorch and TensorFlow, which apply the chain rule for you.
Computing the gradients that gradient descent uses to update the weights.
Networks that read text or other sequences, where the chain runs back through every step of the sequence.

Questions people ask

Why is it called backpropagation?

The error is passed backward through the network, from the loss toward the inputs, one layer at a time. At each step the slopes are multiplied as they pass through.

Is the chain rule the same as backpropagation?

No. The chain rule is a fact of calculus. Backpropagation is the method that applies it to a network and reuses intermediate results, so the work stays efficient.

Why does the sigmoid slope matter?

When $z$ is very large or very small, the sigmoid is nearly flat, so its slope is close to 0. Multiplying by a tiny slope shrinks the gradient, which can slow learning in deep networks.

Do I have to compute these by hand?

Usually not. Libraries compute them automatically. Working through one neuron by hand shows what the library does behind the scenes.

Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.
Sigmoid Function: The sigmoid takes any number, however big or small, and squeezes it into a value between 0 and 1 that can be read as a chance, like 0.8 meaning 80 percent.
Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds the results together with one extra number called the bias, and then uses a simple rule to decide how strong its output should be.