Cross-Entropy Loss

Q: What does ln mean?

The natural log, $\ln$, asks which power of $e$ gives a number. $\ln(1) = 0$, and any number between 0 and 1 gives a negative log. That is why the formula has a minus sign.

In one sentence: Cross-entropy loss gives a small penalty when the model gave the right answer a high probability, and a big penalty when it gave the right answer a small probability.

$L = -\sum_{i=1}^{K} y_i \ln(\hat{y}_i)$

Read it out loud: “L equals minus the sum, for i from 1 up to K, of y sub i times the natural log of y hat sub i. Here L is the loss, K is the number of classes, y sub i is the true label, which is 1 or 0, and y hat sub i is the probability the model gave to class i.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

The natural logarithm $\ln$: The natural logarithm, written $\ln$ and read ‘ln’, asks a question: what power of $e$ gives this number? So $\ln(e) = 1$ , because $e$ to the power 1 is $e$ , and $\ln(1) = 0$ , because $e$ to the power 0 is 1. For a number between 0 and 1, the answer is negative. For example, $\ln(0.5) \approx -0.693$ and $\ln(0.1) \approx -2.303$ . The smaller the number, the more negative its log.
The minus sign in front of a log: A minus sign in front flips the sign of the number. Because a probability is between 0 and 1, its natural log is always negative or zero. The minus sign turns those negative logs into positive penalties. So $-\ln(0.5) = 0.693$ , and $-\ln(0.1) = 2.303$ . This is why the loss is zero or positive, and why a lower loss means a better model.
Labels that are 0 or 1 (one-hot labels): A label is the correct answer written as numbers. In a one-hot label, every class gets a 0 except the correct class, which gets a 1. If the photo really is a cat, the label is $(1, 0, 0)$ for cat, dog and bird. Multiplying by 0 removes a term from a sum, and multiplying by 1 keeps it, so a one-hot label keeps only the correct class in the total.
What a loss is: A loss is one number that says how wrong a model was. Training is the process of making the loss smaller. A loss of 0 would mean a perfect score, and a bigger loss means a bigger mistake. Different tasks use different loss formulas, and cross-entropy is the usual choice when the model picks one class out of several.

What each symbol means

Symbol	Meaning
$L$	The loss, one number that says how wrong the model was on this example. Lower is better, and 0 is a perfect score. When the natural log is used, the unit is the nat, a unit of information used with natural logs, where the bit goes with base-2 logs. Most people read the loss as a plain score with no everyday unit.
$K$	The number of possible classes, such as cat, dog and bird. In the example, $K = 3$ . It is a whole number with no unit, and it sets where the sum sign stops.
$i$	A counter that gives the position of one class in the list, starting at 1. The first class is $i = 1$ , the second is $i = 2$ , and so on up to $K$ . It has no unit and is only a label.
$\sum_{i=1}^{K}$	The sum sign, read ‘add up for every $i$ from 1 to $K$ ’. It goes through every class and totals the terms. Most terms are zero for a one-hot label, so in practice only the correct class’s term remains.
$y_i$	The true label for class $i$ . It is 1 for the correct class and 0 for every other class. It has no unit. It is the answer you know is right, written as numbers so the formula can use it.
$\hat{y}_i$	The probability the model gave to class $i$ . It is a number between 0 and 1, and the hat over the $y$ marks it as the model’s prediction. For each example, the probabilities over all the classes should add up to 1, which is the usual output of softmax.
$\ln$	The natural logarithm, the log with base $e$ . It asks what power of $e$ gives the number inside the bracket. Any number between 0 and 1 gives a negative log, which is why the formula has a minus sign in front of it.

The formula piece by piece

$\ln(\hat{y}_i)$: Start with the probability given to one class and take its natural log. Because $\hat{y}_i$ is between 0 and 1, this log is negative. When the probability is close to 1, the log is close to 0. When the probability is close to 0, the log is a large negative number.
$y_i \ln(\hat{y}_i)$: Multiply the log by the true label. For a wrong class, $y_i$ is 0, so the product is 0 and that class contributes nothing. For the correct class, $y_i$ is 1, so the product is the log itself. This step is how the label chooses which class counts.
$\sum_{i=1}^{K} y_i \ln(\hat{y}_i)$: Add these products for every class in the list. With a one-hot label, every term except the correct one is 0, so the total is the log of the probability given to the correct class.
$L = -\sum_{i=1}^{K} y_i \ln(\hat{y}_i)$: The whole formula in one sentence: the loss equals minus the sum, over all the classes, of the true label times the natural log of the predicted probability, which gives a positive penalty that grows as the probability of the correct answer shrinks.

What the formula is doing, and why

A classifier outputs a probability for each class, such as the probabilities produced by softmax, explained on its own page. The loss asks one question: how much probability did the model give to the answer that was actually right? The loss is then a single number that gets smaller when that probability gets bigger.

Why only the correct class? With a one-hot label, every $y_i$ is 0 except the correct class, where it is 1. In the sum, every term with $y_i = 0$ drops out, so only one term is left: $-\ln(\hat{y}_c)$ , where $c$ is the correct class. The loss is minus the natural log of the probability given to the correct answer, and the wrong answers do not change it directly.

Why the log? The log is what makes the score behave well. When the probability is close to 1, the log is close to 0, so the loss is close to 0. When the probability is close to 0, the log becomes a large negative number, so the loss becomes large. A confident wrong answer is punished hard, and the penalty grows without limit as the probability heads toward 0. A log has this shape, so a tiny probability for the truth costs far more than a moderate one.

Why the minus sign? A probability between 0 and 1 has a negative log, so without the minus sign the loss would be negative. The minus sign turns the negative logs into positive losses, so a lower loss means a better model. Training pushes the loss down, which pushes the probability of the correct answer up.

An everyday picture

Think of a bet where you must put all your money on one horse. If you put a lot of your money on the horse that wins, you lose little. If you put almost nothing on the winner, you lose heavily. Cross-entropy charges the model according to how little it put on the correct answer: a tiny share costs a lot, and the charge rises faster the smaller that share gets.

Worked example: Cat, dog or bird, with a different correct answer

The softmax example gave one photo the probabilities cat $0.659$ , dog $0.242$ and bird $0.099$ . We now ask how much the loss is when each animal is the true answer. Each case uses a one-hot label, so only the correct class has $y = 1$ . The loss has no everyday unit beyond the natural-log scale, so it is shown as a plain number.

Start with the case where the photo really is a cat. The label is $(1, 0, 0)$ , so only the cat term survives in the sum. The other two terms are multiplied by 0 and vanish. The loss is $L = -\ln(0.659)$ .
Compute the log for the cat. The natural log of 0.659 is about $-0.417$ . Because 0.659 is between 0 and 1, its log is negative, and the minus sign in the formula turns it into $L = 0.417$ . This is a small loss, since the model gave the right answer a fairly high probability.
Now the photo really is a dog. The label becomes $(0, 1, 0)$ , so only the dog term survives. The loss is $L = -\ln(0.242) \approx 1.420$ . The model gave the dog only 24.2 percent, so the penalty is larger than for the cat.
Now the photo really is a bird. The label is $(0, 0, 1)$ , and the model gave the bird only 9.9 percent. The loss is $L = -\ln(0.099) \approx 2.313$ . The probability is small, so the log is a large negative number, and the minus sign makes the penalty large and positive.
Compare the cat and bird losses. The bird loss divided by the cat loss is $\frac{2.313}{0.417} \approx 5.5$ . The bird probability is only about 6.7 times smaller than the cat probability, $0.659 / 0.099 \approx 6.7$ , but its loss is about 5.5 times bigger. The penalty grows faster than the probability shrinks, which is the point of using a log.

Answer: When the true answer was cat, the loss is a small 0.417, because the model gave cat 65.9 percent. When the true answer was dog, the loss is 1.420. When the true answer was bird, the model had put only 9.9 percent on it, and the loss jumps to 2.313. The less probability the model gave to the truth, the faster the loss rises.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Dropping the minus sign. Without it the loss would be negative, and training would push the model toward giving the correct answer less probability, which is the wrong direction.
Comparing this loss with a loss computed by a different method without checking. This page uses the natural log. Other log bases give other numbers, so the values are not directly comparable.
Feeding probabilities into a loss function that expects raw scores, or the reverse. Many libraries take raw scores and apply softmax inside, so check the documentation before adding softmax yourself.
Thinking a loss of 0 is normal on real data. A loss of 0 means the model gave probability 1 to the right answer for every case, which is rare.

Where you will meet it

Training image classifiers, where the softmax outputs of the last layer are compared with the true class.
Language models, which score the probability of the correct next word at each position in a text.
Logistic regression for yes-or-no questions, which uses the two-class form of cross-entropy.
Information theory, where cross-entropy measures how many bits or nats are needed to encode data with a model’s probabilities.

Questions people ask

Why not use mean squared error for classes?

Cross-entropy matches how probabilities are scored, and it punishes confident wrong answers much harder than squared error does. Mean squared error is built for numbers, and its page explains the details.

What does ln mean?

The natural log, $\ln$ , asks which power of $e$ gives a number. $\ln(1) = 0$ , and any number between 0 and 1 gives a negative log. That is why the formula has a minus sign.

Is the loss the same as accuracy?

No. Accuracy only asks whether the top answer was right. The loss also counts how confident the model was, so two models with the same accuracy can have different losses.

What is the loss for a yes-or-no question?

It is the two-class form $-[y \ln(\hat{y}) + (1 - y)\ln(1 - \hat{y})]$ . It uses the same idea with two classes instead of many. Here $y$ is 1 for yes and 0 for no, and $\hat{y}$ is the probability of yes.

Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.
Shannon Entropy: Shannon entropy is one number, measured in bits, that says how unpredictable a source of messages is on average: the more predictable it is, the fewer bits its messages need.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.