Kullback-Leibler Divergence

Q: Why is it not symmetric?

Because the average is weighted by $P$. Changing which distribution is the weight changes the answer. In many machine learning uses, $P$ is the data and $Q$ is the model.

In one sentence: KL divergence is a score for how far a guessed set of chances is from the real set of chances: zero means the guess matches the truth, and bigger numbers mean a worse guess.

$D_{\mathrm{KL}}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$

Read it out loud: “D K L of P and Q equals the sum, over every outcome x, of P of x times the natural log of P of x divided by Q of x. The letters D K L stand for Kullback-Leibler divergence. P is the real probability of each outcome, and Q is the guessed probability of each outcome. The natural log, written ln, is a logarithm that uses the number e as its base.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Probability and probability distributions: A probability is a number from 0 to 1 that says how likely an outcome is. A probability distribution lists every possible outcome with its probability, and the probabilities always add up to 1. A fair coin has the distribution heads 0.5 and tails 0.5, and a biased coin might have heads 0.9 and tails 0.1.
Logarithms and the natural log, $\ln$: A logarithm answers the question: what power must we raise a base to, to get this number? The natural log, written $\ln$ , uses the number $e$ , about 2.718, as its base. So $\ln(e) = 1$ and $\ln(1) = 0$ . A number below 1 has a negative log, for example $\ln(0.5) \approx -0.693$ . Logs turn divisions into subtractions, which is why they appear in this formula.
Surprise and information: The surprise of an outcome with probability $P$ is $-\ln P$ . Rare outcomes are more surprising: an outcome with probability 0.1 has a surprise of about 2.30, while an outcome with probability 0.5 has a surprise of about 0.69. Surprise is a way to measure how much information an outcome carries, and the average surprise is the uncertainty in the distribution.
Nats and bits: A nat is the unit of information when the natural log is used, and a bit is the unit when the base-2 log is used. One nat is about 1.443 bits. A bit is the information in one fair yes-or-no answer. Both units measure the same idea with different logarithms, so you can convert between them by multiplying.
Expected value, the weighted average: An expected value is a weighted average. You multiply each value by its probability and add the results. KL divergence is an expected value of the log ratio, with the probabilities $P(x)$ as the weights, so outcomes that happen often count more than outcomes that rarely happen.

What each symbol means

Symbol	Meaning
$D_{\mathrm{KL}}(P \,\\|\, Q)$	The Kullback-Leibler divergence of $Q$ from $P$ : a score that is never negative. It is zero only when the two distributions are identical. Its unit is the nat when the natural log is used, and the bit when the base-2 log is used.
$P$	The true or reference distribution: the real probability of each outcome. It is a list of numbers between 0 and 1 that add up to 1, with no unit. A fair coin gives $P(\text{heads}) = 0.5$ .
$Q$	The guessed or model distribution: the probabilities that a model or a guess gives to the same outcomes. Like $P$ , it is a list of numbers between 0 and 1 that add up to 1. A model that guesses heads 0.9 of the time gives $Q(\text{heads}) = 0.9$ .
$x$	One possible outcome, such as heads or tails for a coin toss. Outcomes are the things the probability distribution gives a chance to. It has no unit, and it is a label for one outcome.
$P(x)$	The probability of outcome $x$ under the real distribution $P$ . It is a number from 0 to 1 with no unit. For a fair coin, $P(\text{heads}) = 0.5$ , which means heads comes up about half the time.
$Q(x)$	The guessed probability of outcome $x$ under $Q$ . It must be greater than 0 wherever $P(x)$ is greater than 0, because the ratio $P(x)/Q(x)$ would divide by zero otherwise. It has no unit.
$\log$	The logarithm. This page uses the natural log, $\ln$ , which gives nats. A base-2 log would give bits instead. It has no unit of its own, because the unit comes from the choice of base. For example, $\ln(1) = 0$ .
$\sum_x$	Add up over every possible outcome $x$ . It has no unit of its own. It totals the weighted log ratios across all outcomes, so every outcome contributes one term to the answer.

The formula piece by piece

$\frac{P(x)}{Q(x)}$: The ratio of the two probabilities for one outcome. It is bigger than 1 when $Q$ gives the outcome too small a chance, and smaller than 1 when $Q$ gives it too large a chance. A ratio of exactly 1 means the guess matches the truth for that outcome.
$\log \frac{P(x)}{Q(x)}$: The log of that ratio, which is the difference in surprise between the two descriptions. It is zero when the ratio is 1, positive when the guess is too small for the outcome, and negative when the guess is too large.
$P(x) \log \frac{P(x)}{Q(x)}$: Weight that difference by the real probability of the outcome. Outcomes that happen often count more, and rare outcomes count less, because the average is taken over what really happens.
$D_{\mathrm{KL}}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$: KL divergence is the average, weighted by the true probabilities, of how much more surprised you are when you use the guess $Q$ instead of the truth $P$ .

What the formula is doing, and why

A probability distribution gives a probability to every outcome, and the probabilities add up to 1. Suppose the real world follows $P$ , but you describe it with $Q$ . KL divergence measures the cost of that mismatch: the extra surprise, on average, from using $Q$ when $P$ is the truth. A value of zero means there is no extra surprise at all.

What each term does. The factor $P(x)$ says how often outcome $x$ really happens, so common outcomes count more. The ratio $P(x)/Q(x)$ compares the two probabilities for that outcome. The logarithm of that ratio is the difference in surprise between the two descriptions, and the sum averages it over all the outcomes.

Why the log? A log turns a ratio into a difference: $\ln\frac{P}{Q} = \ln P - \ln Q$ . Surprise is measured as $-\ln$ of a probability, so the difference of the two logs is exactly the difference in surprise between the two descriptions. That is why the log appears in the formula.

Never negative, but not symmetric. The result is never negative, and it is exactly zero only when $Q$ matches $P$ for every outcome. It is not symmetric, though: $D_{KL}(P \| Q)$ and $D_{KL}(Q \| P)$ are usually different numbers, because the average is weighted by $P$ . For this reason KL divergence is not a true distance.

Units. The base of the logarithm only changes the unit. The natural log gives nats, and the base-2 log gives bits. One nat is about 1.443 bits, so you multiply a value in nats by 1.443 to get bits. Use one unit consistently when you compare numbers.

An everyday picture

Think of a weather forecast as a guess $Q$ for how often each kind of day happens, and the real climate as $P$ . KL divergence is the average extra surprise you feel by planning with the forecast’s odds instead of the real ones. The closer the guess matches the real odds, the smaller the surprise, and a perfect match gives zero.

Worked example: A fair coin and a biased guess

The real coin is fair: $P(\text{heads}) = 0.5$ and $P(\text{tails}) = 0.5$ . A model guesses $Q(\text{heads}) = 0.9$ and $Q(\text{tails}) = 0.1$ . The probabilities have no unit. We use the natural log, so the answer is in nats. All numbers are made up for this example.

Set up the terms. KL divergence adds one term per outcome. For heads, the term is $P(\text{heads}) \ln \frac{P(\text{heads})}{Q(\text{heads})}$ , which uses the real chance 0.5 and the guess 0.9. The formula compares the two numbers for each outcome, then weights the result by the real chance of that outcome.
Heads term. The ratio is $\frac{0.5}{0.9} \approx 0.5556$ , which is less than 1 because the model guessed heads too often. Its natural log is $\ln(0.5556) \approx -0.5878$ , a negative number. Multiplying by the real chance 0.5 gives $0.5 \times (-0.5878) = -0.2939$ . The negative sign tells us the model over-guessed heads.
Tails term. The ratio is $\frac{0.5}{0.1} = 5$ , which is bigger than 1 because the model guessed tails too rarely. Its natural log is $\ln 5 \approx 1.6094$ . Multiplying by 0.5 gives $0.5 \times 1.6094 = 0.8047$ . This positive term is larger than the heads term, because the tails mistake is bigger: the model gave tails only 0.1.
Add the two terms. $D_{KL}(P \| Q) = -0.2939 + 0.8047 = 0.5108$ nats. The total is positive, so the guess is worse than perfect. To read it in bits, multiply by 1.443, which gives about $0.737$ bits. A bit is the smaller unit, so the number is bigger in bits.
Swap the roles to check the asymmetry. $D_{KL}(Q \| P) = 0.9 \ln \frac{0.9}{0.5} + 0.1 \ln \frac{0.1}{0.5} = 0.5290 - 0.1609 = 0.3681$ nats. The answer differs from 0.5108 because now the weights are the guess’s probabilities. That is why KL divergence is not symmetric.

Answer: The divergence of the guess $Q$ from the truth $P$ is about 0.511 nats, or about 0.737 bits. The reverse direction gives about 0.368 nats. The two numbers differ, so KL divergence is not symmetric and is not a distance.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Treating KL divergence as a distance. It is not symmetric, so $D_{KL}(P \| Q)$ and $D_{KL}(Q \| P)$ can differ, and it does not obey the triangle inequality that real distances follow.
Mixing up the order. The first distribution is the one the average is weighted by. Swapping the arguments answers a different question, as the example shows.
Putting a zero in $Q$ where $P$ is not zero. The term $\ln(P/0)$ is infinite, so the divergence becomes infinite. Models often add a tiny amount to every probability to avoid this.
Comparing a value in nats with a value in bits. They differ by a factor of about 1.443, so the units must match before you compare two numbers.

Where you will meet it

Training variational autoencoders, where a KL term keeps the learned distribution close to a simple reference distribution.
Knowledge distillation, where a small student model is trained to match the output probabilities of a larger teacher model.
Fine-tuning large language models, where a KL penalty often keeps the tuned model close to a reference model.
Information theory, where it measures the extra bits needed to encode data when you use the wrong probabilities.

Questions people ask

Why is it not symmetric?

Because the average is weighted by $P$ . Changing which distribution is the weight changes the answer. In many machine learning uses, $P$ is the data and $Q$ is the model.

Can KL divergence be negative?

No. It is always 0 or more, and it is 0 only when $P$ and $Q$ are the same. This follows from Gibbs’ inequality, a standard result in information theory.

What is the Jensen-Shannon divergence?

It averages the KL divergences from $P$ and from $Q$ to their common midpoint, a mixture of the two. The result is symmetric and always finite, and its square root is a true distance.

How does KL divergence relate to cross-entropy?

Cross-entropy equals the entropy of $P$ plus $D_{KL}(P \| Q)$ . The entropy of $P$ is the average surprise of outcomes drawn from $P$ , and it is a fixed number when $P$ is fixed. So minimizing cross-entropy is the same as minimizing the KL divergence.

Cross-Entropy Loss: Cross-entropy loss gives a small penalty when the model gave the right answer a high probability, and a big penalty when it gave the right answer a small probability.
Shannon Entropy: Shannon entropy is one number, measured in bits, that says how unpredictable a source of messages is on average: the more predictable it is, the fewer bits its messages need.
Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.