# Positional Encoding > Positional encoding gives each word a pattern of waves that says where it sits in the sentence, so a model that reads all the words at once still knows their order. Page: https://formulabro.yunusemrevurgun.com/ai-ml/positional-encoding/ Category: AI & Machine Learning Published: 2026-10-22 Updated: 2026-10-22 ## Formula $$ \text{PE}_{(pos,\,2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right) \qquad \text{PE}_{(pos,\,2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right) $$ Read it out loud: "PE at position pos, index 2i, equals the sine of pos divided by 10000 raised to the power 2i over d. PE at index 2i plus 1 equals the cosine of the same angle. PE is the positional encoding. The letter pos is the position of the word, counting from 0. The letter i counts pairs of numbers, starting from 0, and the letter d is the size of each list of numbers. The sine and cosine are wave functions, and 10000 is a fixed number from the original paper." ## Before you start - **Vectors and embeddings**: A **vector** is an ordered list of numbers, treated as one object. A **word embedding** is a vector that stands for a word's meaning, with $d$ numbers in it. The position encoding is also a list of $d$ numbers, so the two lists have the same length and can be added together number by number. - **Sine and cosine**: **Sine**, written $\sin$, and **cosine**, written $\cos$, are wave functions. Give them an angle and they return a number between $-1$ and $1$. As the angle grows, they rise and fall smoothly, and together they trace a circle. For example, $\sin 0 = 0$ and $\cos 0 = 1$. - **Radians and pi**: An angle in this formula is measured in **radians**, not degrees. A full turn is $2\pi \approx 6.283$ radians, so one radian is about 57.3 degrees. The Greek letter $\pi$, called pi, is about 3.14159. The angle 2 radians is about 115 degrees, and 0.02 radians is about 1.1 degrees. - **Powers, like $10000^{2i/d}$**: A **power** says how many times to multiply a number by itself, and a fraction in the power can mean a root. Any number to the power 0 is 1, so $10000^{0} = 1$. The power $10000^{1/2}$ is the square root of 10000, which is 100, because $100 \times 100 = 10000$. In the formula the power $\frac{2i}{d}$ is a fraction, so the divisor grows as the pair index grows. - **Frequency of a wave**: A wave **repeats** over and over. Its **frequency** is how often it repeats, and a fast wave completes many repeats in a short span. In this formula each pair of numbers is one wave. Pairs with a small index repeat quickly, and pairs with a large index repeat slowly, so together the pairs describe position at many different scales. - **Transformer and attention**: A **transformer** is a neural network design for sequences such as sentences. **Attention** lets each word look at the other words and decide how much each one matters. A transformer reads all the words at the same time, rather than one after another, so it needs position information to know the order. - **Pairs of places and the index $i$**: The list of $d$ numbers is split into pairs. Places 0 and 1 form the first pair, places 2 and 3 form the second pair, and so on. The index $i$ counts the pairs, starting at 0. The pair with index $i$ uses the places $2i$ and $2i+1$. So with $d = 4$ there are two pairs, with $i = 0$ and $i = 1$. More notation help: https://formulabro.yunusemrevurgun.com/math-basics/ ## What each symbol means - $\text{PE}_{(pos,\,2i)}$: The **encoding value** at position $pos$ in an even-numbered place, $2i$. It is a plain number with no unit. It always lies between $-1$ and $1$, because it comes from a sine, and it is added to the word embedding. - $\text{PE}_{(pos,\,2i+1)}$: The encoding value at the same position in the next, odd-numbered place, $2i+1$. It comes from a cosine, so it also lies between $-1$ and $1$. Each even and odd pair shares the same frequency. - $pos$: The **position** of the word in the sentence, counting from 0 for the first word. So the second word has $pos = 1$. It is a count with no unit. - $i$: The **pair index**, counting from 0 up to $\frac{d}{2} - 1$. A small $i$ gives a fast wave, and a large $i$ gives a slow wave. It has no unit. - $d$: The **size** of the encoding: the number of places in each list, which equals the size of the word embedding. It must be even, so that the numbers come in pairs. The original paper used $d = 512$. - $10000$: A fixed constant from the original paper. It sets how slow the slowest wave is: the wavelengths run from $2\pi$ up to about $10000 \times 2\pi$. It has no unit. - $\sin$: The **sine function**, a wave that turns an angle into a number from $-1$ to $1$. Its input is an angle measured in radians. - $\cos$: The **cosine function**, a wave like the sine but shifted by a quarter turn. It also gives numbers from $-1$ to $1$, and its input is the same angle in radians. ## The formula piece by piece - $pos$: The position of the word, counting from 0. It is the only thing that changes from one word to the next, so it decides which wave values each word receives. - $\frac{2i}{d}$: A fraction that describes which pair we are using, and it lies between 0 and 1. For $d = 4$, pair $i = 0$ gives $\frac{0}{4} = 0$, and pair $i = 1$ gives $\frac{2}{4} = 0.5$. It sets how slow the pair's wave is. - $10000^{2i/d}$: The divisor for this pair, a power of 10000. With exponent 0 the divisor is 1, and with exponent 0.5 it is 100. A bigger divisor means the position changes the angle more slowly, so the wave is slower. - $\frac{pos}{10000^{2i/d}}$: The angle, in radians, for this word and this pair. The position is divided by the divisor, so a bigger position gives a bigger angle, and a bigger divisor gives a smaller angle. - $\sin\left(\frac{pos}{10000^{2i/d}}\right)$: The sine of that angle. It is the value for the even place, $2i$. Because sine stays between $-1$ and $1$, the value never grows, however large the position becomes. - $\text{PE}_{(pos,\,2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right) \qquad \text{PE}_{(pos,\,2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$: **Each word's position code has $d$ numbers. In each pair, the even place holds the sine of the angle, and the odd place holds the cosine of the same angle.** ## What the formula is doing, and why A transformer reads all the words at the same time. Attention on its own does not care about order, so *dog bites man* and *man bites dog* would look like the same set of vectors. Positional encoding fixes this by adding a position signal to each word's vector, so the same word in a different place gets a different vector. The input is a position $pos$ and the size $d$. The output is a list of $d$ numbers for that position. The numbers come in pairs. The places $2i$ and $2i+1$ share one frequency, set by the divisor $10000^{2i/d}$. The first number of each pair is a sine, and the second is a cosine, of the same angle. So each pair moves around a circle as the position grows. **Why many frequencies?** The frequencies form a series that runs from fast to slow. Pairs with a small $i$ change quickly from one position to the next, which tells neighboring words apart. Pairs with a large $i$ change slowly, which keeps track of position across long distances. Read all the pairs together, and each position has its own pattern of numbers. **Why sine and cosine, and why add?** A sine alone repeats, so two different angles can give the same value. Pairing it with a cosine places each pair on a circle, so the angle can be read without confusion. The original paper adds the encoding to the embedding instead of appending it, so the vector keeps its size $d$, which the rest of the model expects. The authors also hoped the pattern would help the model handle sentences longer than those it saw in training. That was a hope, not a proven result. **When it applies.** Position codes like this one are needed by any transformer that reads all positions at once. Other models give order in different ways: some learn their position vectors during training, and others build position into the attention step itself. The formula gives a value for any position, but very long inputs can still cause trouble in practice. ## An everyday picture Think of a clock with several hands that turn at different speeds. The fastest hand tells you small differences in time, and the slowest hand tells you roughly where you are in a long day. Read all the hands together and the time is unique. Positional encoding works the same way: the fast waves tell neighboring positions apart, and the slow waves place words far down the sentence. ## Worked example: Encoding position 2 in a four-number vector Use $d = 4$, so each word gets four numbers in two pairs. Find the encoding for position $pos = 2$, which is the third word, because positions start counting at 0. 1. **Find each pair's divisor.** For pair $i = 0$, the power is $\frac{2 \times 0}{4} = 0$, and $10000^{0} = 1$. For pair $i = 1$, the power is $\frac{2 \times 1}{4} = 0.5$, and $10000^{0.5}$ is the square root of 10000, which is $100$, because $100 \times 100 = 10000$. 2. **Work out the first pair.** The angle is $\frac{2}{1} = 2$ radians, about 115 degrees. Its sine is $\text{PE}_{(2,0)} = \sin(2) \approx 0.9093$ for the even place. Its cosine is $\text{PE}_{(2,1)} = \cos(2) \approx -0.4161$ for the odd place. 3. **Work out the second pair.** The angle is $\frac{2}{100} = 0.02$ radians, only about 1.1 degrees. Its sine is $\text{PE}_{(2,2)} = \sin(0.02) \approx 0.0200$, and its cosine is $\text{PE}_{(2,3)} = \cos(0.02) \approx 0.9998$. The slow pair has barely moved away from its starting value, which is what a long-range signal looks like. 4. **Stack the four values.** The encoding for position 2 is $(0.9093,\ -0.4161,\ 0.0200,\ 0.9998)$. This list is added, number by number, to the embedding of the third word in the sentence, so the word's vector carries its position. 5. **Compare with the next position.** At position $pos = 3$, the first pair's angle is 3 radians, which gives $\sin 3 \approx 0.1411$ and $\cos 3 \approx -0.9900$. The fast pair has changed a lot. The slow pair only moves to about $0.0300$ and $0.9996$. Comparing the two positions shows how neighboring words are told apart. **Answer:** The position vector for position 2 is **(0.9093, -0.4161, 0.0200, 0.9998)**. The first pair changes quickly from one position to the next, so it separates neighboring words. The second pair changes slowly, so it carries a longer-range signal. ## Common mistakes - Adding a plain position number, such as 2, to the embedding. A single number is not kept within fixed limits, and it has no pattern across the places of the vector, so the formula spreads position over many waves instead. - Using an odd embedding size. The sine and cosine values come in pairs that share a frequency, so $d$ should be even. - Assuming sinusoidal encodings are the only option. Many models learn their position vectors during training, and others build position into the attention step itself, for example with rotary encodings. - Believing the formula alone limits how long the input can be. It gives a value for any position, but a model may still handle very long inputs poorly. ## Where you will meet it - The original transformer for machine translation in Vaswani et al. (2017), which added these sine and cosine vectors to its input word embeddings. - Early transformer language models and other sequence models that need order information, since attention alone treats its inputs as an unordered set. - Newer position methods, such as learned position vectors and rotary encodings, which many later models use instead of the fixed sine and cosine pattern. - Any transformer that reads a sequence all at once, such as models for text, audio or time series, wherever the order of the steps matters. ## Questions people ask ### Why use sine and cosine instead of plain numbers? Sine and cosine always stay between $-1$ and $1$, so the values never grow with the position. Mixing many frequencies also gives each position its own pattern. ### Why does each pair have both a sine and a cosine? A sine alone repeats, so two different angles can give the same value. Pairing it with a cosine places each pair on a circle, so the angle, and therefore the position, can be read without confusion. ### Is the encoding learned during training? Not in the original paper. The formula is fixed. Other models learn their position vectors as part of training. ### Does the encoding change the meaning of a word? It shifts the word's vector by a position-dependent amount. The model learns to use both the meaning part and the position part when it pays attention to other words. ## Related formulas - [Scaled Dot-Product Attention](https://formulabro.yunusemrevurgun.com/ai-ml/scaled-dot-product-attention/) - [Softmax](https://formulabro.yunusemrevurgun.com/ai-ml/softmax/) - [Perplexity](https://formulabro.yunusemrevurgun.com/ai-ml/perplexity/) ## Sources - [Vaswani et al., Attention Is All You Need (arXiv:1706.03762)](https://arxiv.org/abs/1706.03762) - [Transformer (deep learning), Wikipedia](https://en.wikipedia.org/wiki/Transformer_(deep_learning)) From Formula Bro (https://formulabro.yunusemrevurgun.com/), a side project developed by yunusemrevurgun.com.