Scaled Dot-Product Attention

In one sentence: Attention lets a model look at every item in a list and decide how much each one matters for the item it is working on, giving more weight to the items that match best.

$\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V$

Read it out loud: “Attention of Q, K and V equals softmax of the quantity Q times K transposed, divided by the square root of d sub k, all multiplied by V. Q, K and V are three tables of numbers: queries, keys and values. K transposed is the keys table turned on its side, and d sub k is how many numbers are in each key.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Vectors: lists of numbers: A vector is a short list of numbers that is treated as one object, such as $(1, 0)$ or $(3, 4)$ . Each number in the list is a component. In attention, each item in a sentence is described by a vector: a list of, say, 64 numbers that stand for what that item means or looks for. A vector can also be drawn as an arrow with a length and a direction, which is why a list of numbers can point in some direction.
The dot product $\cdot$: The dot product of two vectors multiplies their matching numbers and adds the results. For example, $(1, 2) \cdot (3, 4) = 1 \times 3 + 2 \times 4 = 11$ . A big dot product means the two vectors point in similar directions. A dot product of 0 means they are at a right angle, so they have nothing in common. A negative dot product means they point apart.
Tables of numbers, rows and the transpose $K^{\top}$: A matrix is a table of numbers with rows and columns. Here $Q$ , $K$ and $V$ are tables with one row per item in the list. The small raised $\top$ means transpose, which flips the table on its side so that rows become columns. If $K$ has a row for each item, then $K^{\top}$ has a column for each item, so every row of $Q$ can be dotted with every row of $K$ in one step.
Square roots, like $\sqrt{d_k}$: The square root of a number is the value that, multiplied by itself, gives the number. So $\sqrt{9} = 3$ , because $3 \times 3 = 9$ . A number that is not a perfect square gives a decimal, such as $\sqrt{2} \approx 1.414$ . The square root symbol $\sqrt{\ }$ is the inverse of squaring, so dividing by $\sqrt{d_k}$ is the same as scaling by a moderate amount when $d_k$ grows.
Weights that add up to 1, and the exponential $e^{x}$: A weight here is a share of the attention, a number between 0 and 1. The weights for one query always add up to exactly 1, so they can be read as percentages of the total focus. The exponential $e^{x}$ means Euler’s number $e$ , about 2.718, raised to the power $x$ . It turns any score into a positive number, and bigger scores become much bigger. Softmax uses this to turn raw scores into weights that add up to 1.

What each symbol means

Symbol	Meaning
$\operatorname{Attention}(Q, K, V)$	The output of the attention step: for each query, a blend of all the values, where each value is weighted by how well that query matches its key. It is a table of vectors with one row per item, and it has no unit.
$Q$	The queries, a table with one row per item. Each row is a vector describing what that item is looking for. The numbers in it have no fixed unit, because the model learns what they mean during training.
$K$	The keys, a table with one row per item. Each row describes what that item offers, like a label on a box. A query is matched against every key to find the best matches. Like $Q$ , its numbers have no fixed unit.
$V$	The values, a table with one row per item. Each row holds the content that gets passed along when that item is attended to. Once the weights are known, the output is a weighted mix of these rows, so values are what the output is built from.
$K^{\top}$	The keys turned on their side, the transpose of $K$ . Turning the table on its side lines up the keys so that each query’s row can be dotted with each key’s column in one step, which gives every pairwise match score at once.
$d_k$	The length of each key vector, meaning how many numbers it holds. If each key is the list $(1, 0)$ , then $d_k = 2$ . It is a whole number with no unit, and it sets how much the match scores are scaled down.
$\sqrt{d_k}$	The square root of $d_k$ . Dividing the match scores by this number keeps them from growing too large when the vectors are long. In the example, $\sqrt{2} \approx 1.414$ . It has no unit.
$\operatorname{softmax}$	The softmax rule, which turns a list of scores into positive weights that add up to exactly 1. It is applied to each row of the match scores, so every query gets its own set of weights, one for each item in the list.

The formula piece by piece

$Q K^{\top}$: Start with the matching step. Each query row is dotted with each key row, which gives one match score for every pair of items. The result is a table of raw scores, with a big score wherever a query and a key point in similar directions.
$\frac{Q K^{\top}}{\sqrt{d_k}}$: Divide every raw score by the square root of the key length. This is the scaling step. It keeps the scores at a sensible size, because dot products of long vectors would otherwise grow large and make the weights unhelpfully sharp.
$\operatorname{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right)$: Apply softmax to each row of the scaled scores. Each row becomes a set of positive weights that add up to 1, so each item has a share of attention for every other item. These weights are the attention.
$\operatorname{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V$: Multiply the weights by the values and add them up. Each output row is a weighted mix of the value rows, where items that matched well contribute more of their content, and poor matches contribute little.
$\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V$: The whole formula in one sentence: attention of the queries, keys and values equals the softmax of the scaled match scores between queries and keys, applied to the values, so each item’s output is a weighted blend of all the items’ values.

What the formula is doing, and why

Attention works on a list of items, such as the words in a sentence. Each item is turned into three vectors. The query says what the item is looking for. The key says what the item offers. The value holds the content the item contributes. All three come from the same words through weight matrices that the model learns during training. Three roles let the model separate what to look for from what to advertise and what to pass along.

Why dot products? The first step is matching. Take the query of one item and take the dot product with the key of every item in the list. A dot product is large when the two vectors point in similar directions, and small or negative when they point apart. That makes the dot product a simple measure of how well a query fits a key. The result is one raw score for each pair of items.

Why divide by $\sqrt{d_k}$ ? Dot products grow as the vectors get longer, because more pairs of numbers are added together. Long keys would give huge scores. Huge scores push softmax to put almost all of its weight on one item, and learning slows down. Dividing by $\sqrt{d_k}$ keeps the scores at a sensible size, so the weights stay spread out enough to learn from.

Why softmax and a weighted sum? Softmax turns each row of scores into weights that are positive and add up to 1. The output for each query is the weighted sum of the values. Items that match the query well contribute more of their value, and items that match poorly contribute little. The output is a blend, not a copy of any single value.

An everyday picture

Think of searching a library. Your question is the query. Each book’s catalog card is its key, and the book’s contents are its value. You compare your question with every card and give each book a score for how well it matches. Then you read a mix of the books in proportion to those scores, mostly from the best matches and a little from the rest.

Worked example: A toy search with one query and two items

Imagine a tiny system where each item has a two-number key and a two-number value. The first number of each vector means ‘sweet’ and the second means ‘savory’. One query $q = (1, 0)$ says ‘I want something sweet’. The two keys are $k_1 = (1, 0)$ for an apple and $k_2 = (0, 1)$ for a piece of cheese. The values are $v_1 = (10, 0)$ and $v_2 = (0, 10)$ , the content each item passes along. Each key has length $d_k = 2$ . All numbers are made up for this example and have no unit.

Dot the query with each key. The query is matched against every key, so we compute $q \cdot k_1 = 1 \times 1 + 0 \times 0 = 1$ for the apple and $q \cdot k_2 = 1 \times 0 + 0 \times 1 = 0$ for the cheese. The apple matches the sweet query, so its score is higher.
Divide by $\sqrt{d_k}$ . The square root of 2 is about 1.414, so the scaled scores are $1 / 1.414 \approx 0.707$ for the apple and $0 / 1.414 = 0$ for the cheese. Scaling keeps these numbers small, which is the reason for the division in the formula.
Apply softmax to the two scaled scores. Raise $e$ to each score: $e^{0.707} \approx 2.028$ for the apple and $e^{0} = 1$ for the cheese. These add up to $2.028 + 1 = 3.028$ . Dividing each by the total gives the weights $\frac{2.028}{3.028} \approx 0.670$ for the apple and $\frac{1}{3.028} \approx 0.330$ for the cheese, and they add up to 1.
Take the weighted sum of the values. Multiply each value by its weight and add the results: $0.670 \times (10, 0) + 0.330 \times (0, 10)$ . The first part is $(6.70, 0)$ and the second part is $(0, 3.30)$ . Adding them gives $(6.70, 3.30)$ , the output for the query.

Answer: The output is (6.70, 3.30). It is mostly the apple’s value, because the query matched the apple’s key better. The cheese still gets about a third of the weight, so its value is not ignored. The weights were $0.670$ and $0.330$ , which add up to 1.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Leaving out the division by $\sqrt{d_k}$ . With long vectors the scores become huge, and the weights collapse onto a single item, so the model stops looking at the others.
Mixing up the roles of Q, K and V. Queries and keys decide how much each item attends to the others, and values are the content that gets copied into the output.
Thinking attention reads the items in order. Plain attention looks at all items at once, so word order has to be added separately, for example with positional encodings.
Expecting the output to be a copy of the best-matching value. The output is a blend of all the values, weighted by the scores.

Where you will meet it

The original transformer translation model in Vaswani et al. (2017), the paper that introduced the transformer.
Large language models, where many attention layers let each word use context from the words that came before it.
Vision transformers, which apply attention to small patches of an image.
Speech recognition systems built on transformer architectures, which attend across the whole audio sequence.

Questions people ask

Why are there three vectors instead of one?

Separate roles let the model learn what to look for, what to advertise and what to pass along. A single vector would have to do all three jobs at once.

Why is it called self-attention?

When the queries, keys and values all come from the same sequence, the items attend to each other. That is called self-attention. In cross-attention, the queries come from one sequence while the keys and values come from another.

Is attention the same as softmax?

No. Softmax is one step inside attention. Attention also includes the dot products that score the matches, the scaling and the final weighted sum.

Why do long inputs make attention expensive?

The score table has one entry for every pair of items, so its size grows quickly as the list gets longer. That is one reason long inputs cost so much for transformers.

Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.
Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds the results together with one extra number called the bias, and then uses a simple rule to decide how strong its output should be.
Cross-Entropy Loss: Cross-entropy loss gives a small penalty when the model gave the right answer a high probability, and a big penalty when it gave the right answer a small probability.