Batch Normalization

In one sentence: Batch normalization rescales a group of values so they average out to zero with a steady spread, then lets the network learn its own final scale and shift.

$\hat x_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \qquad y_i = \gamma \, \hat x_i + \beta$

Read it out loud: “x-hat sub i equals x sub i minus mu sub B, divided by the square root of sigma sub B squared plus epsilon. Then y sub i equals gamma times x-hat sub i, plus beta. Here x sub i is one input value, and mu sub B is the average of the batch. Sigma sub B squared is the variance, a measure of spread, and epsilon is a tiny safety number. Gamma and beta are learned numbers. The Greek letters are mu, said myoo; sigma, said SIG-muh; epsilon, said EP-sih-lon; gamma, said GAM-uh; and beta, said BAY-tuh.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Mini-batch: A mini-batch is a small group of examples that the network processes together in one training step, such as 32 photos. Each neuron in a layer produces one value per example, so one neuron gives a whole list of values, one for each photo in the batch. Batch normalization works on the values from that group, not on the whole data set.
The mean, $\mu_B$ , the average of a batch: The mean of a list is its average: add the values and divide by how many there are. The mean of 2, 4, 6 and 8 is $(2 + 4 + 6 + 8) / 4 = 5$ . The Greek letter mu, $\mu$ (said myoo), is the usual symbol for a mean. The subscript $B$ says that the mean is taken over the mini-batch $B$ .
The variance, $\sigma_B^{2}$ , and the spread: The variance measures spread. Find the distance of each value from the mean, square each distance so that negatives count as positive, and average the squares. For the numbers 2, 4, 6 and 8 the distances from 5 are $-3, -1, 1, 3$ , their squares are $9, 1, 1, 9$ , and the average of the squares is 5. The Greek letter sigma, $\sigma$ (said SIG-muh), is the usual symbol for spread, so $\sigma^{2}$ is the variance.
The square root, $\sqrt{\cdot}$ , and standard deviation: The square root $\sqrt{x}$ is the number that, multiplied by itself, gives $x$ . Since $\sqrt{5} \approx 2.236$ , and $2.236 \times 2.236 \approx 5$ , the square root of the variance 5 is about 2.236. That value is the standard deviation $\sigma$ , the typical distance from the mean, in the same unit as the data.
Powers of ten, like $10^{-5}$ , and epsilon: A number such as $10^{-5}$ is a power of ten: the exponent says how many times to divide by 10. So $10^{-5} = 1 / 100000 = 0.00001$ , a decimal point followed by four zeros and then a 1. The Greek letter epsilon, $\epsilon$ (said EP-sih-lon), is a tiny positive number added inside the square root. It keeps the division safe when the spread is almost zero.
Learned scale $\gamma$ and shift $\beta$: $\gamma$ (gamma, said GAM-uh) and $\beta$ (beta, said BAY-tuh) are two numbers that the network learns during training, in the same way that it learns its weights. $\gamma$ multiplies the normalized values to set their spread, and $\beta$ is added to set their center. Many implementations start with $\gamma = 1$ and $\beta = 0$ , which leave the normalized values unchanged at the beginning.

What each symbol means

Symbol	Meaning
$x_i$	One input value from the mini-batch, for example one neuron’s value for one example in the batch. The subscript $i$ says which example it is. It has whatever unit the feature has, and in this example it is unitless.
$\mu_B$	The mean of the values in the mini-batch $B$ , their average. It has the same unit as $x_i$ . For the values 2, 4, 6 and 8 the mean is 5.
$\sigma_B^{2}$	The variance of the batch values: the average of the squared distances from the mean. It measures spread. Its unit is the square of the unit of $x_i$ , so it can look large even when the spread is modest.
$\epsilon$	A tiny positive constant, about $10^{-5}$ in many implementations (the example uses 0.001), added inside the square root so that the division never divides by zero. It has the same unit as the variance and is chosen to be very small.
$\sqrt{\sigma_B^{2} + \epsilon}$	The square root of the variance plus epsilon, which is the typical spread of the batch in the same unit as $x_i$ . Dividing by it turns the spread into a standard size close to 1.
$\hat{x}_i$	The normalized value: the centered and rescaled $x_i$ , with mean 0 and variance close to 1 across the batch. It has no unit, because the units cancel when we divide by the spread.
$\gamma$	The learned scale factor, a number that the network chooses during training. It has no unit. It sets how spread out the layer’s outputs are, and many implementations start it at 1.
$\beta$	The learned shift, a number that the network chooses during training. It has the same unit as the output of the layer. It sets the center of the outputs, and many implementations start it at 0.
$y_i$	The output for example $i$ after the scale and shift, which is passed on to the next layer. It has the same unit as the output of the layer.

The formula piece by piece

$x_i - \mu_B$: Centering: subtract the batch average from one value. Values above the average become positive, values below it become negative, and the average itself becomes zero. In the example, the value 8 minus the mean 5 gives 3.
$\sqrt{\sigma_B^{2} + \epsilon}$: The spread of the batch, with the tiny epsilon added so that it is never zero. In the example, $\sqrt{5 + 0.001} \approx 2.2363$ .
$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^{2} + \epsilon}}$: Normalizing: the centered value divided by the spread, so the batch ends up with mean 0 and a spread close to 1. In the example, the value 8 becomes $3 / 2.2363 \approx 1.3415$ .
$y_i = \gamma \hat{x}_i + \beta$: Scaling and shifting: multiply the normalized value by the learned $\gamma$ , which sets the spread, and add the learned $\beta$ , which sets the center. In the example, $\gamma = 2$ doubles the spread and $\beta = 1$ moves the center up to 1.
$\hat x_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \qquad y_i = \gamma \, \hat x_i + \beta$: The whole rule in one sentence: each value is centered on the batch average and divided by the batch spread, then multiplied by gamma and shifted by beta, so the layer can learn its own final spread and center.

What the formula is doing, and why

What goes in and what comes out. Batch normalization takes a group of values for one feature or one neuron, taken from the examples in the mini-batch $B$ . It returns new values that are centered and scaled. Inside a deep network, the values that reach a layer keep shifting as the layers before it learn, so each layer receives values whose average and spread keep moving.

Why center and divide? The mean $\mu_B$ is the average of the batch, and the variance $\sigma_B^{2}$ is the average squared distance from that mean. Subtracting $\mu_B$ moves the center to zero. Dividing by the square root of the variance sets the spread to about 1. The tiny $\epsilon$ sits inside the square root, so that a batch whose values are all equal, which has zero spread, does not cause a division by zero.

Why learn gamma and beta? Forcing every layer to a mean of 0 and a spread of 1 can throw away useful information. So each layer also gets two learned numbers, $\gamma$ and $\beta$ . The output $y_i = \gamma \hat{x}_i + \beta$ can take whatever spread and center training finds useful. If $\gamma = 1$ and $\beta = 0$ , the output equals the normalized values. If training chooses other numbers, the layer can even undo the normalization completely.

Where it applies, and prediction time. Batch normalization works best with reasonably large batches, because the mean and variance from a tiny batch are noisy. The paper that introduced it reported faster training, and the method became common in image models. When a model makes predictions one example at a time, there is no batch to average, so the layer uses running averages of the mean and variance that were collected during training.

An everyday picture

Think of a class whose test scores drift upward from one test to the next. Before comparing the results, a teacher rescales each test so the average is 0 and the typical spread is 1, which puts every test on the same footing. The learned $\gamma$ and $\beta$ are like the teacher then choosing the grading scale that works best for the class.

Worked example: Four values in one mini-batch

Four examples go through one neuron in a mini-batch, and each gives one value: $x = (2, 4, 6, 8)$ . Use $\epsilon = 0.001$ , and the learned values $\gamma = 2$ and $\beta = 1$ . The values are unitless, as in a toy example.

Find the mean of the batch. The mean tells us the center of the group. We add the four values, $2 + 4 + 6 + 8 = 20$ , and divide by 4, so $\mu_B = 20 / 4 = 5$ . The center of this batch is 5.
Find the variance of the batch. The variance is the average squared distance from the mean. The distances from 5 are $-3, -1, 1$ and $3$ . Squaring gives $9, 1, 1$ and $9$ , because squaring makes every distance positive. Their average is $(9 + 1 + 1 + 9) / 4 = 5$ , so $\sigma_B^{2} = 5$ .
Compute the denominator. The denominator is the spread with epsilon added to keep it safe: $\sqrt{5 + 0.001} = \sqrt{5.001} \approx 2.2363$ . The epsilon changes the answer by only a tiny amount, which is what it is designed to do.
Normalize each value. Subtract the mean from each value and divide by the denominator. That gives $\hat{x} = (2 - 5) / 2.2363, \; (4 - 5) / 2.2363, \; (6 - 5) / 2.2363, \; (8 - 5) / 2.2363$ , which is about $(-1.3415, -0.4472, 0.4472, 1.3415)$ . The new values average to 0. Their variance is $5 / 5.001 \approx 0.9998$ , which is very close to 1, as intended.
Scale and shift. Each normalized value is multiplied by $\gamma = 2$ , and then $\beta = 1$ is added: $y = 2\hat{x} + 1$ . The outputs are about $(-1.6830, 0.1057, 1.8943, 3.6830)$ . The scale of 2 doubles the spread, and the shift of 1 moves the center from 0 up to 1.
Check the mean of the outputs. The mean of the outputs is $\gamma \times 0 + \beta = 1$ , because the normalized values average to zero. Adding the outputs gives $-1.6830 + 0.1057 + 1.8943 + 3.6830 = 4.0000$ , and $4.0 / 4 = 1.0$ , which equals $\beta$ .

Answer: The four outputs have a mean of $\beta = 1$ , and their spread is set by $\gamma = 2$ . The batch supplied the mean and the variance, and the learned $\gamma$ and $\beta$ set the final spread and center.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Taking the statistics across the features of one example. Batch normalization averages across the batch, for each feature. Layer normalization is the version that averages across the features of one example.
Using a batch of one or two examples. The mean and variance are then very noisy, and the normalization can make training worse.
Forgetting the running averages at prediction time. A single example should use the stored running averages, not a batch of one.
Dropping epsilon when the values in a batch are all equal. The variance is then zero, and the division by the square root of zero fails.

Where you will meet it

Image recognition networks, which sort photos into categories, where a normalization step often sits after each layer.
Residual networks, a common family of image networks, which often use batch normalization between their layers.
Models trained with large mini-batches, where the batch statistics are most reliable.
Sequence models, such as text models, which often use layer normalization instead, because batch statistics are awkward when inputs vary in length.

Questions people ask

What is a mini-batch?

A small group of examples that the network processes together in one training step, such as 32 or 64 photos. The batch statistics come from that group.

Why does batch normalization need a batch?

Its mean and variance come from the examples in the batch. With one example there is no spread to measure, so at prediction time the layer switches to stored running averages.

Does batch normalization replace the learning rate?

No. The learning rate, the step size of training, still matters. Batch normalization changes the scale of the values each layer sees, which often allows a larger learning rate to work well.

What do gamma and beta do?

They are learned along with the other weights. They let each layer choose its own spread and center, so normalizing does not force one fixed scale.

Layer Normalization: Layer normalization takes the list of numbers that describe one example, makes their average 0 and their spread about 1, so the numbers stay in a steady range whatever other examples are nearby.
Dropout: Dropout randomly turns off some of a network’s neurons during each practice round and boosts the ones left on, so the network cannot lean on any single neuron.
Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds the results together with one extra number called the bias, and then uses a simple rule to decide how strong its output should be.