Positional Encoding

In one sentence: Positional encoding gives each word a pattern of waves that says where it sits in the sentence, so a model that reads all the words at once still knows their order.

Read it out loud: “PE at position pos, index 2i, equals the sine of pos divided by 10000 raised to the power 2i over d. PE at index 2i plus 1 equals the cosine of the same angle. PE is the positional encoding. The letter pos is the position of the word, counting from 0. The letter i counts pairs of numbers, starting from 0, and the letter d is the size of each list of numbers. The sine and cosine are wave functions, and 10000 is a fixed number from the original paper.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Vectors and embeddings

A vector is an ordered list of numbers, treated as one object. A word embedding is a vector that stands for a word’s meaning, with $d$ numbers in it. The position encoding is also a list of $d$ numbers, so the two lists have the same length and can be added together number by number.

Sine and cosine

Sine, written $\sin$, and cosine, written $\cos$, are wave functions. Give them an angle and they return a number between $-1$ and $1$. As the angle grows, they rise and fall smoothly, and together they trace a circle. For example, $\sin 0 = 0$ and $\cos 0 = 1$.

Radians and pi

An angle in this formula is measured in radians, not degrees. A full turn is $2\pi \approx 6.283$ radians, so one radian is about 57.3 degrees. The Greek letter $\pi$, called pi, is about 3.14159. The angle 2 radians is about 115 degrees, and 0.02 radians is about 1.1 degrees.

Powers, like $10000^{2i/d}$

A power says how many times to multiply a number by itself, and a fraction in the power can mean a root. Any number to the power 0 is 1, so $10000^{0} = 1$. The power $10000^{1/2}$ is the square root of 10000, which is 100, because $100 \times 100 = 10000$. In the formula the power $\frac{2i}{d}$ is a fraction, so the divisor grows as the pair index grows.

Frequency of a wave

A wave repeats over and over. Its frequency is how often it repeats, and a fast wave completes many repeats in a short span. In this formula each pair of numbers is one wave. Pairs with a small index repeat quickly, and pairs with a large index repeat slowly, so together the pairs describe position at many different scales.

Transformer and attention

A transformer is a neural network design for sequences such as sentences. Attention lets each word look at the other words and decide how much each one matters. A transformer reads all the words at the same time, rather than one after another, so it needs position information to know the order.

Pairs of places and the index $i$

The list of $d$ numbers is split into pairs. Places 0 and 1 form the first pair, places 2 and 3 form the second pair, and so on. The index $i$ counts the pairs, starting at 0. The pair with index $i$ uses the places $2i$ and $2i+1$. So with $d = 4$ there are two pairs, with $i = 0$ and $i = 1$.

What each symbol means

The formula piece by piece

$pos$

The position of the word, counting from 0. It is the only thing that changes from one word to the next, so it decides which wave values each word receives.

$\frac{2i}{d}$

A fraction that describes which pair we are using, and it lies between 0 and 1. For $d = 4$, pair $i = 0$ gives $\frac{0}{4} = 0$, and pair $i = 1$ gives $\frac{2}{4} = 0.5$. It sets how slow the pair’s wave is.

$10000^{2i/d}$

The divisor for this pair, a power of 10000. With exponent 0 the divisor is 1, and with exponent 0.5 it is 100. A bigger divisor means the position changes the angle more slowly, so the wave is slower.

$\frac{pos}{10000^{2i/d}}$

The angle, in radians, for this word and this pair. The position is divided by the divisor, so a bigger position gives a bigger angle, and a bigger divisor gives a smaller angle.

$\sin\left(\frac{pos}{10000^{2i/d}}\right)$

The sine of that angle. It is the value for the even place, $2i$. Because sine stays between $-1$ and $1$, the value never grows, however large the position becomes.

$\text{PE}_{(pos,\,2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right) \qquad \text{PE}_{(pos,\,2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$

Each word’s position code has $d$ numbers. In each pair, the even place holds the sine of the angle, and the odd place holds the cosine of the same angle.

What the formula is doing, and why

A transformer reads all the words at the same time. Attention on its own does not care about order, so dog bites man and man bites dog would look like the same set of vectors. Positional encoding fixes this by adding a position signal to each word’s vector, so the same word in a different place gets a different vector.

The input is a position $pos$ and the size $d$. The output is a list of $d$ numbers for that position. The numbers come in pairs. The places $2i$ and $2i+1$ share one frequency, set by the divisor $10000^{2i/d}$. The first number of each pair is a sine, and the second is a cosine, of the same angle. So each pair moves around a circle as the position grows.

Why many frequencies? The frequencies form a series that runs from fast to slow. Pairs with a small $i$ change quickly from one position to the next, which tells neighboring words apart. Pairs with a large $i$ change slowly, which keeps track of position across long distances. Read all the pairs together, and each position has its own pattern of numbers.

Why sine and cosine, and why add? A sine alone repeats, so two different angles can give the same value. Pairing it with a cosine places each pair on a circle, so the angle can be read without confusion. The original paper adds the encoding to the embedding instead of appending it, so the vector keeps its size $d$, which the rest of the model expects. The authors also hoped the pattern would help the model handle sentences longer than those it saw in training. That was a hope, not a proven result.

When it applies. Position codes like this one are needed by any transformer that reads all positions at once. Other models give order in different ways: some learn their position vectors during training, and others build position into the attention step itself. The formula gives a value for any position, but very long inputs can still cause trouble in practice.

An everyday picture

Think of a clock with several hands that turn at different speeds. The fastest hand tells you small differences in time, and the slowest hand tells you roughly where you are in a long day. Read all the hands together and the time is unique. Positional encoding works the same way: the fast waves tell neighboring positions apart, and the slow waves place words far down the sentence.

Worked example: Encoding position 2 in a four-number vector

Use $d = 4$, so each word gets four numbers in two pairs. Find the encoding for position $pos = 2$, which is the third word, because positions start counting at 0.

1. Find each pair’s divisor. For pair $i = 0$, the power is $\frac{2 \times 0}{4} = 0$, and $10000^{0} = 1$. For pair $i = 1$, the power is $\frac{2 \times 1}{4} = 0.5$, and $10000^{0.5}$ is the square root of 10000, which is $100$, because $100 \times 100 = 10000$.
2. Work out the first pair. The angle is $\frac{2}{1} = 2$ radians, about 115 degrees. Its sine is $\text{PE}_{(2,0)} = \sin(2) \approx 0.9093$ for the even place. Its cosine is $\text{PE}_{(2,1)} = \cos(2) \approx -0.4161$ for the odd place.
3. Work out the second pair. The angle is $\frac{2}{100} = 0.02$ radians, only about 1.1 degrees. Its sine is $\text{PE}_{(2,2)} = \sin(0.02) \approx 0.0200$, and its cosine is $\text{PE}_{(2,3)} = \cos(0.02) \approx 0.9998$. The slow pair has barely moved away from its starting value, which is what a long-range signal looks like.
4. Stack the four values. The encoding for position 2 is $(0.9093,\ -0.4161,\ 0.0200,\ 0.9998)$. This list is added, number by number, to the embedding of the third word in the sentence, so the word’s vector carries its position.
5. Compare with the next position. At position $pos = 3$, the first pair’s angle is 3 radians, which gives $\sin 3 \approx 0.1411$ and $\cos 3 \approx -0.9900$. The fast pair has changed a lot. The slow pair only moves to about $0.0300$ and $0.9996$. Comparing the two positions shows how neighboring words are told apart.

Answer: The position vector for position 2 is (0.9093, -0.4161, 0.0200, 0.9998). The first pair changes quickly from one position to the next, so it separates neighboring words. The second pair changes slowly, so it carries a longer-range signal.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

• Adding a plain position number, such as 2, to the embedding. A single number is not kept within fixed limits, and it has no pattern across the places of the vector, so the formula spreads position over many waves instead.
• Using an odd embedding size. The sine and cosine values come in pairs that share a frequency, so $d$ should be even.
• Assuming sinusoidal encodings are the only option. Many models learn their position vectors during training, and others build position into the attention step itself, for example with rotary encodings.
• Believing the formula alone limits how long the input can be. It gives a value for any position, but a model may still handle very long inputs poorly.

Where you will meet it

• The original transformer for machine translation in Vaswani et al. (2017), which added these sine and cosine vectors to its input word embeddings.
• Early transformer language models and other sequence models that need order information, since attention alone treats its inputs as an unordered set.
• Newer position methods, such as learned position vectors and rotary encodings, which many later models use instead of the fixed sine and cosine pattern.
• Any transformer that reads a sequence all at once, such as models for text, audio or time series, wherever the order of the steps matters.

Questions people ask

Why use sine and cosine instead of plain numbers?

Sine and cosine always stay between $-1$ and $1$, so the values never grow with the position. Mixing many frequencies also gives each position its own pattern.

Why does each pair have both a sine and a cosine?

A sine alone repeats, so two different angles can give the same value. Pairing it with a cosine places each pair on a circle, so the angle, and therefore the position, can be read without confusion.

Is the encoding learned during training?

Not in the original paper. The formula is fixed. Other models learn their position vectors as part of training.

Does the encoding change the meaning of a word?

It shifts the word’s vector by a position-dependent amount. The model learns to use both the meaning part and the position part when it pays attention to other words.

Related formulas

• Scaled Dot-Product Attention: Attention lets a model look at every item in a list and decide how much each one matters for the item it is working on, giving more weight to the items that match best.
• Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.
• Perplexity: Perplexity counts how many words a language model is torn between, on average, when it guesses the next word. Lower is better.

Sources

1. Vaswani et al., Attention Is All You Need (arXiv:1706.03762)
2. Transformer (deep learning), Wikipedia