Perplexity

Q: What does a perplexity of 10 mean?

On average, the model was as unsure as if it had to choose evenly among 10 words at each step. Lower values mean the model gave higher probabilities to the words that actually came next.

Q: Why use the exponential at the end?

The average of the logs is in nats and is hard to picture. The exponential turns it back into a count of effective choices, which is easier to read.

Q: Is perplexity the same as cross-entropy?

They are closely linked. Perplexity is $e$ raised to the average cross-entropy, measured in nats. If you measure in bits, perplexity is 2 raised to the cross-entropy in bits.

Q: Is a lower perplexity always better?

It is better at predicting the test text. It does not show that the answers are true, useful or safe, so it is only one part of judging a model.

In one sentence: Perplexity counts how many words a language model is torn between, on average, when it guesses the next word. Lower is better.

$\text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \ln p(w_i \mid w_{<i})\right)$

Read it out loud: “PPL equals e raised to the power of the negative of one over N, times the sum, for i from 1 to N, of the natural log of p of w sub i given w sub less than i. PPL is perplexity. The letter e is about 2.718. N is the number of tokens, which are the pieces of text the model reads. The letter w sub i is the actual token at position i, and w sub less than i means all the tokens before position i. The vertical bar is read as given. The capital sigma means add up, and ln means the natural logarithm.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Language model and tokens: A language model is a program that guesses the next word of a text from the words before it. It reads text in small units called tokens. A token is usually a whole word or a piece of a word. A test text is a piece of text that is held back and used only for checking. The number $N$ is how many tokens the test text contains, and perplexity is measured on the test text.
Probability given earlier words, $p(w_i \mid w_{<i})$: $p(w_i \mid w_{<i})$ is read “the probability of word $w_i$ given the words before it.” The vertical bar means given, and $w_{<i}$ means all the tokens before position $i$ . So the expression is the chance the model gave to the real word at position $i$ , after it had read everything earlier. The value is a fraction from 0 to 1, so 0.5 means a 50-50 chance.
The natural logarithm, $\ln$ , and the number $e$: The number $e$ is about 2.71828. The natural logarithm, written $\ln$ , asks: what power of $e$ gives this number? So $\ln(e^{x}) = x$ , and $\ln 1 = 0$ , because $e$ to the power 0 is 1. A number between 0 and 1 gives a negative log: $\ln 0.5 \approx -0.693$ , because $e$ to the power $-0.693$ is 0.5.
Surprise and cross-entropy: The surprise of a token is $-\ln p$ , where $p$ is the probability the model gave that token. A likely word gives a small surprise, close to 0, and an unlikely word gives a large surprise. Cross-entropy is the average surprise over all the tokens in the test text. It is measured in nats.
Nats and bits, the units of surprise: A nat is the unit of surprise when we use the natural log. A bit is the unit when we use the logarithm with base 2 instead. One bit equals about 0.693 nats. The unit does not change which model is better, but to get back to a perplexity you raise $e$ to the power of a value in nats, and you raise 2 to the power of a value in bits.
Averages and the sum sign: Dividing a total by a count gives an average. The capital sigma, $\sum$ , means add up, and $\sum_{i=1}^{N}$ adds the terms for $i = 1$ up to $i = N$ . So $\frac{1}{N} \sum_{i=1}^{N} x_i$ means: add up all $N$ values, then divide by $N$ .
Scientific notation, like $2.5 \times 10^{-3}$: Very small numbers are often written as a number times a power of ten. The small raised number, the exponent, says how many times to multiply by 10. A minus sign means divide by 10 that many times, so $10^{-3} = \frac{1}{10 \times 10 \times 10} = 0.001$ . Then $2.5 \times 10^{-3}$ means 2.5 thousandths, which is 0.0025. This way of writing numbers is called scientific notation.

What each symbol means

Symbol	Meaning
$\text{PPL}$	Perplexity, a plain number with no unit. Lower values mean the model predicted the text better. A perplexity of $k$ means the model was, on average, as unsure as if it had chosen evenly among $k$ words.
$N$	The number of tokens in the test text, a plain count with no unit. A longer test text gives an average over more tokens.
$i$	The position of one token in the text, counting from 1 to $N$ . It is a label, so it has no unit.
$w_i$	The actual token at position $i$ , the piece of text that really appears there and that the model tries to predict. It is a piece of text, so it has no unit.
$w_{<i}$	All the tokens that come before position $i$ , which is the context the model has read so far. The symbol $<$ means less than, so the subscript reads as the positions less than $i$ .
$p(w_i \mid w_{<i})$	The probability the model gives to the actual token $w_i$ after reading the tokens before it. It is a fraction from 0 to 1 with no unit.
$\ln$	The natural logarithm, the power of $e$ needed to produce a number. It is negative for any number between 0 and 1, so the log of a probability is always negative. Its result is measured in nats.
$\exp$	The exponential function, $\exp(x) = e^{x}$ , which means $e$ raised to the power $x$ . It turns the average surprise back into a count-like number. Its result is always positive.
$\sum_{i=1}^{N}$	The add-up sign over every token position $i$ from 1 to $N$ . It adds the surprise of each token into one total, which is then divided by $N$ to get the average.

The formula piece by piece

$p(w_i \mid w_{<i})$: The chance the model gave to the real next word. It is a fraction from 0 to 1. A value of 0.5 means the model thought that word had a 50-50 chance, and a value of 0.1 means only one chance in ten.
$\ln p(w_i \mid w_{<i})$: The natural log of that chance. Because the chance is below 1, its log is negative. A likely word gives a value close to 0, and an unlikely word gives a value far below 0.
$-\ln p(w_i \mid w_{<i})$: The surprise of one token. The minus sign flips the negative log into a positive number, so every token adds a positive amount of surprise. A rare word, with a very small chance, adds a large surprise.
$\sum_{i=1}^{N} -\ln p(w_i \mid w_{<i})$: The total surprise over the whole test text. We add the surprise of every token from position 1 to position $N$ , so a longer text gives a bigger total.
$-\frac{1}{N}\sum_{i=1}^{N} \ln p(w_i \mid w_{<i})$: The average surprise per token, which is the cross-entropy measured in nats. Dividing the total by $N$ turns it into an average, so texts of different lengths can be compared fairly.
$\text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \ln p(w_i \mid w_{<i})\right)$: Perplexity equals $e$ raised to the power of the average surprise per token. The exponential turns that average back into a count of how many words the model was torn between.
$\text{PPL} = \left(\prod_{i=1}^{N} p(w_i \mid w_{<i})\right)^{-1/N}$: A second standard form. The capital pi, $\prod$ , means multiply all the terms, so this is the product of every probability, raised to the power $-\frac{1}{N}$ . It gives the same number, because multiplying probabilities and averaging their logs are two views of one calculation.

What the formula is doing, and why

The input is a test text and the probability the model gave to each real token. The output is one number, the perplexity. For every token, the model gave a chance to the word that really came next. Perplexity combines those chances over the whole text into one score that describes how well the model predicted the text.

Why use the logarithm? The chance of a whole text is the product of all the single-token chances, and a product of many small fractions is hard to handle. Taking logs turns that product into a sum, and a sum can be averaged. The log of each chance is negative, so the minus sign in front turns each value into a positive surprise. Averaging the surprise gives the cross-entropy.

Why the exponential at the end? The average surprise in nats is hard to picture. The exponential changes it back into a count. A perplexity of 4 means the model was, on average, as unsure as if it had to choose evenly among four words each time. A perplexity of 1 would mean the model was certain of every next word.

Why is lower better? A model that gives the real next words higher probabilities has a smaller average surprise, so it has a smaller perplexity. Perplexity only compares models fairly when they are scored on the same test text, with the same way of splitting text into tokens. A different splitting rule gives a different number of tokens $N$ , and then the two numbers cannot be compared.

When it applies. Perplexity measures how well a model predicts a text. It can be used on any language model that gives a probability for each token. It does not say whether the answers are true, helpful or safe, so it is one measure among several.

An everyday picture

Think of a quiz with several doors at every step, where one door hides the right answer. A model that knows the answer is sure of one door. A model that is torn between four doors at each step has a perplexity of about four. Perplexity counts how many doors, on average, the model still has open before each word arrives.

Worked example: A four-token test text with mixed confidence

A word predictor reads a test text of 4 tokens. For each token, it gave the word that really came next these probabilities: $0.2$ , $0.5$ , $0.25$ and $0.1$ . We want one number that describes how well it predicted this short text.

Write down the four probabilities. The predictor gave the real next word a chance of $0.2$ , then $0.5$ , then $0.25$ , then $0.1$ . These are fractions between 0 and 1, so $0.5$ means a 50-50 chance and $0.1$ means one chance in ten. A low chance means the word was surprising.
Find the surprise of each word with the natural log. $\ln 0.2 \approx -1.6094$ , $\ln 0.5 \approx -0.6931$ , $\ln 0.25 \approx -1.3863$ and $\ln 0.1 \approx -2.3026$ . Each value is negative because every chance is below 1. The most surprising word, with chance $0.1$ , has the most negative log.
Add the four values. Adding logs is the same as multiplying the chances, and that is why logs are used here: $-1.6094 - 0.6931 - 1.3863 - 2.3026 \approx -5.9915$ . This sum is the total of the logs for the whole four-token text.
Average and flip the sign. Divide the total by the number of tokens, $N = 4$ , to get the average, then flip the sign to get the surprise: $-\frac{-5.9915}{4} \approx 1.4979$ nats. This average surprise per token is the cross-entropy, measured in nats.
Turn the average back into a count with $e$ . Raise $e$ , about 2.718, to the power 1.4979: $\text{PPL} = e^{1.4979} \approx 4.472$ . Raising $e$ to the average undoes the natural log, and the result is a count of effective choices per token.
Check the answer with the product of the chances. Multiply the four probabilities: $0.2 \times 0.5 \times 0.25 \times 0.1 = 0.0025$ , which is $2.5 \times 10^{-3}$ . Then raise it to the power $-\frac{1}{4}$ . That means the fourth root of $\frac{1}{0.0025} = 400$ , and the fourth root of 400 is about 4.472, the same answer.
Say what the number means. A perplexity of about 4.47 means that, on average, the model was as unsure as if it had to choose evenly among about four and a half equally likely words at each step. The product check gives the same number, which confirms that the average is right.

Answer: The perplexity is about 4.47. On average, the model was as unsure as if it had to choose among roughly four and a half equally likely words at each step. The product check gives the same number, which shows what the average means.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Leaving out the minus sign. The log of a probability is negative, so without the minus the average comes out negative, and the exponential falls below 1.
Comparing perplexity across models that split text into tokens in different ways. A different splitting rule gives a different number of tokens $N$ , so the numbers are not comparable.
Mixing units. If you measure surprise in bits, with base-2 logs, the exponential must use 2 as its base too. Using $e$ with values in bits gives a wrong answer.
Thinking that lower perplexity means a more helpful or truthful chatbot. Perplexity only measures how well the model predicted a test text.

Where you will meet it

Tracking the training of language models, where the perplexity on held-out text is watched as a single progress number.
Comparing language models on the same test text, where a lower perplexity means better next-word prediction.
Speech recognition and machine translation, where a language model rescores candidate sentences and prefers the ones with lower perplexity.
Text compression, where the probabilities a model predicts decide how many bits a coder needs for each character or word.

Questions people ask

What does a perplexity of 10 mean?