Layer Normalization

In one sentence: Layer normalization takes the list of numbers that describe one example, makes their average 0 and their spread about 1, so the numbers stay in a steady range whatever other examples are nearby.

$\hat x_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \qquad y_i = \gamma_i \, \hat x_i + \beta_i$

Read it out loud: “x-hat sub i equals x sub i minus mu, all divided by the square root of sigma squared plus epsilon. Then y sub i equals gamma sub i times x-hat sub i, plus beta sub i. Read mu as mew, sigma as SIG-muh, epsilon as EP-sih-lon, gamma as GAM-uh and beta as BAY-tuh. Here x sub i is one number in the list, mu is the average of the list, sigma squared measures how spread out the list is, and x-hat is the centered and rescaled number.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Greek letters and subscripts, like $\mu$ , $\sigma$ , $\epsilon$ , $\gamma$ , $\beta$ and $x_i$: Greek letters are extra names for quantities, because the Latin alphabet runs out of letters. $\mu$ is read mew and usually stands for an average. $\sigma$ is read sigma, and $\sigma^2$ is read sigma squared. $\epsilon$ is epsilon, $\gamma$ is gamma and $\beta$ is beta. A small number under a letter, such as $x_i$ , is a subscript that picks one item from a list. $x_1$ is the first item, and $x_i$ means the item at position $i$ , for whichever position is meant.
Average, variance and square root: The average, or mean, of a list is its sum divided by how many numbers there are. The variance measures how spread out the numbers are: take each number’s distance from the average, square each distance, and average those squares. Its unit is the square of the original unit. The square root, written $\sqrt{\cdot}$ , undoes squaring, so $\sqrt{9} = 3$ because $3 \times 3 = 9$ .
Feature and vector: A feature is one number that describes an example, such as one test score or one measurement. A vector is a list of features treated as one object. In a transformer, the network design behind modern language models, each word is one vector, and its features are numbers that together describe that word’s meaning inside the model. Layer normalization works on one such list at a time.
Neural network, parameters and learned values: A neural network is a model made of many small calculators called neurons, and it learns its numbers from examples. The numbers it learns are called parameters or weights, and training adjusts them until the answers improve. The scale $\gamma_i$ and the shift $\beta_i$ are parameters of this kind. The model starts from 1 and 0, then changes them to whatever works best.
Hats on letters, like $\hat{x}_i$: A small hat over a letter, such as $\hat{x}_i$ (read x-hat sub i), marks a new version of the same quantity. Here $\hat{x}_i$ is the normalized version of $x_i$ : the value after the average has been subtracted and the result divided by the spread. The original $x_i$ is kept for the calculation, and the hat tells you which version you are reading.
Approximately equal, $\approx$ , and rounding: The sign $\approx$ means approximately equal. Many square roots have decimals that never end, so $\sqrt{125} \approx 11.18$ is written after rounding to a few digits. Rounding changes only the last digits. When you check a step with a calculator, you should get the same first three or four digits, which is enough to confirm the answer.

What each symbol means

Symbol	Meaning
$x_i$	One feature of the example: a single number in its list of features. For example, one of a student’s four test scores, measured in points out of 100. It has the unit of the feature, so a score is measured in points.
$\mu$	The average of all the features of this one example. It is in the same unit as $x_i$ , so if the scores are in points, $\mu$ is also in points. For a student whose four scores are 60, 70, 80 and 90, $\mu = 75$ points, the middle of the scores.
$\sigma^2$	The variance of the features of this one example: the average of the squared distances from $\mu$ . Its unit is points squared, which is hard to picture, so read it as a measure of spread. A small variance means the scores sit close together, and a large one means they are far apart.
$\epsilon$	A tiny positive number, such as 0.001, added so that the division never divides by zero. It has no unit. It is far smaller than a typical variance, so it changes the result very little, and it only matters when every feature is almost the same.
$\hat x_i$	The normalized feature: the original value after subtracting the average and dividing by the spread. It has no unit, because points divided by points cancel. Across the features of one example, the normalized values always average to 0, and their spread is close to 1.
$\gamma_i$	A learned scale for feature $i$ . It multiplies the normalized value and starts at 1, which leaves the value unchanged. It has no unit. Each feature has its own scale, so the model can make some features larger and others smaller as training goes on.
$\beta_i$	A learned shift for feature $i$ , added after the scaling. It starts at 0, which adds nothing. It has the same unit as the output, which is no unit here. Each feature has its own shift, which lets the model move the center of each feature away from zero if that helps.
$y_i$	The final output for feature $i$ , after the scale and shift are applied. It has no unit in this example. The layer passes these numbers on to the next part of the network, so each output is the input of the next calculation.

The formula piece by piece

$x_i$: Start with the input: one feature, one number from the example’s list, such as one test score. Every other piece of the formula is built from these numbers together with their average and spread.
$x_i - \mu$: Subtract the average. After this step the numbers are centered on zero: a positive result means the feature is above the average of its example, and a negative result means it is below. The relative sizes of the features do not change, only their common offset is removed.
$\sigma^2 + \epsilon$: Build the safe spread. The variance measures how spread out the features are, and adding the tiny number epsilon means the bottom of the next fraction can never be zero, even when all the features are equal.
$\frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}$: Divide the centered value by the spread. Taking the square root turns the variance back into the original unit, so the result is a plain number. It tells you how many spreads a feature sits above or below the average of its example.
$\gamma_i \, \hat x_i + \beta_i$: Scale and shift. Multiply the normalized value by the learned scale, then add the learned shift. Starting values of 1 and 0 leave the normalized value unchanged, so the model begins with a plain normalization.
$\hat x_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \qquad y_i = \gamma_i \, \hat x_i + \beta_i$: Layer normalization centers and rescales each example’s own numbers, and then the model learns a scale and a shift for each feature, so every feature stays in a steady range.

What the formula is doing, and why

Layer normalization takes one example at a time and rescales its own numbers. The example might be a word in a sentence, described by a list of features. The formula does two things. First it subtracts the average $\mu$ of those features, so the list is centered on zero. Then it divides by the spread, $\sqrt{\sigma^2 + \epsilon}$ , so the typical size of the numbers is about 1. The result is the normalized value $\hat x_i$ . The model then applies a learned scale and shift to get the final output $y_i$ .

Why subtract the average? In a deep network the numbers passed between layers can drift far above or below zero. Large offsets make learning unstable. Subtracting $\mu$ moves the whole list so that its average is exactly 0. Nothing about the relative sizes of the features changes, because every feature moves by the same amount.

Why divide by the spread? If the numbers are very large, the next layer receives huge inputs; if they are tiny, it receives weak inputs. Dividing by the square root of the variance puts the list on a common scale. The variance is in squared units, so the square root is needed to bring the spread back to the unit of the features. Epsilon is added so that a list whose features are all equal, with zero spread, does not cause a division by zero.

Why per example? Batch normalization, the older method, takes its averages over a batch of many examples. Layer normalization only uses the features of the one example in front of it. So the result for an example is the same whether it runs alone or in a group. This matters in transformers, because sentences have different lengths and each word is handled on its own.

Why learn a scale and a shift? Centering and scaling force every example into the same average and spread, which may not suit every feature. The learned scale $\gamma_i$ and shift $\beta_i$ let the model restore whatever center and spread it needs. They start at 1 and 0. Layer normalization needs a list with at least two features to be useful, because a list of a single number has zero spread, and then the output is only the shift $\beta$ .

An everyday picture

Think of a music app that evens out the loudness of one song by itself. It measures the average loudness of that song and how much its volume jumps around, then rescales the whole song using only those two numbers. It does not compare the song with the rest of the playlist, so the song sounds the same no matter which songs were played before or after it. Layer normalization treats one example’s features in the same way.

Worked example: Four test scores from one student

One student has four test scores, $x = (60, 70, 80, 90)$ , each in points out of 100. Think of these four numbers as the four features of one example. Use $\epsilon = 0.001$ and the starting values $\gamma_i = 1$ and $\beta_i = 0$ , so the scale does nothing and the shift adds nothing at the start. The scores are in points, so the variance is in points squared.

Find the average score. The average, $\mu$ , tells us the center of this student’s four scores. We add the scores and divide by how many there are: $\mu = \frac{60 + 70 + 80 + 90}{4} = \frac{300}{4} = 75$ points.
Measure each score’s distance from the average. Next we see how far each score sits from 75, with a plus sign if it is above and a minus sign if it is below. The distances are $60 - 75 = -15$ , $70 - 75 = -5$ , $80 - 75 = 5$ and $90 - 75 = 15$ points. They add up to zero, which is always true when distances are measured from the average.
Square the distances and average them. Squaring makes every distance positive, so the signs stop mattering for the spread. The squares are $225$ , $25$ , $25$ and $225$ points squared, and their sum is 500. Dividing by four gives the variance: $\sigma^2 = \frac{500}{4} = 125$ points squared.
Build the bottom of the fraction. We add the tiny epsilon to the variance and take the square root: $\sqrt{125 + 0.001} = \sqrt{125.001} \approx 11.1804$ points. The extra 0.001 changes the root by less than a ten-thousandth of a point, so the epsilon has almost no effect here, which is the point of choosing it small.
Normalize each score. Divide each centered score by 11.1804. For the first score, $\frac{-15}{11.1804} \approx -1.3416$ . The unit cancels, because points divided by points leaves a plain number. The other three become $-0.4472$ , $0.4472$ and $1.3416$ , in the same order as the original scores.
Apply the scale and shift. Because $\gamma_i = 1$ and $\beta_i = 0$ , each output is $y_i = 1 \times \hat x_i + 0 = \hat x_i$ . Multiplying by 1 and adding 0 change nothing, so at the start the outputs equal the normalized values. A trained model would replace these starting values with learned ones.
Check the result. The four outputs should average to 0 and have a variance close to 1. Their sum is $-1.3416 - 0.4472 + 0.4472 + 1.3416 = 0$ , so the average is 0. The squares of the normalized values add up to about 4.00, so the variance is about $4.00 / 4 = 1$ . Those two facts are what the normalization is designed to produce.

Answer: The normalized values are $(-1.3416,\ -0.4472,\ 0.4472,\ 1.3416)$ . The best score sits about 1.34 spreads above this student’s average, and the worst score sits about 1.34 spreads below it. Because the average and spread came from this one student alone, adding other students to a batch would not change these numbers.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Averaging over the batch instead of over the features of one example. That is batch normalization, and the result then changes when the other examples in the batch change.
Taking the average across the words of a sentence instead of across the features of each word. Layer normalization works on one vector at a time, so the average and spread come from a single word’s features.
Expecting one shared scale and shift for all features. Each feature has its own $\gamma_i$ and $\beta_i$ , so the model can treat features differently.
Dropping $\epsilon$ when every feature of a vector is equal. The variance is then zero, and the division would be by zero, so the tiny $\epsilon$ keeps the bottom of the fraction above zero.

Where you will meet it

Transformer models for language, such as the encoder and decoder blocks inside many large language models.
Vision transformers and other attention-based image models, which use the same per-example rescaling.
Recurrent neural networks, the setting the original 2016 paper studied, where batch averages are awkward to use.
Models that handle one example at a time, such as live prediction, where no batch of examples exists to take averages from.

Questions people ask

Why do transformers use layer normalization instead of batch normalization?

Batch normalization depends on the other examples in its batch, and sentences have different lengths. Layer normalization works on one vector at a time, so its result does not depend on the other examples.

How is it different from batch normalization in practice?

Batch normalization keeps running averages from training so it can be used when predicting one example. Layer normalization computes its averages from the example itself, every time, so it needs no running averages at all.

What is a feature here?

One number in the list that describes an example. For a word in a sentence, the list might hold many numbers, each one a learned hint about the word’s meaning inside the model.

Does layer normalization change the order of the features?

No. Each feature keeps its position in the list. The average and spread only set the center and size of the whole list, so the first feature is still the first feature after normalization.

Batch Normalization: Batch normalization rescales a group of values so they average out to zero with a steady spread, then lets the network learn its own final scale and shift.
Scaled Dot-Product Attention: Attention lets a model look at every item in a list and decide how much each one matters for the item it is working on, giving more weight to the items that match best.
Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.