Shannon Entropy

In one sentence: Shannon entropy is one number, in bits, that says how unpredictable a source is. The more predictable it is, the less space its messages need.

$H = -\sum_{i=1}^{n} p_i \log_2 p_i$

Read it out loud: “H equals minus the sum, for every outcome i from 1 to n, of p sub i times the base-2 logarithm of p sub i. The answer is measured in bits.”

What each symbol means

Symbol	Meaning
$H$	The entropy of the source, in bits. One bit is the information in one fair yes-or-no answer.
$n$	How many different possible outcomes there are, such as 2 for a coin or 6 for a die.
$i$	Which outcome we are looking at, counted from 1 to $n$ .
$p_i$	The probability of outcome $i$ : a number from 0 to 1. All the $p_i$ together add up to 1.
$\log_2 p_i$	The base-2 logarithm of the probability. It is zero or negative for any probability at most 1, which is why the formula has a minus sign in front.
$-\sum_{i=1}^{n}$	“Add up the terms for every outcome, then flip the sign.” Flipping the sign makes the final answer zero or positive.

What the formula is doing

Shannon entropy answers one simple question: on average, how surprised should you be by the next message from a source? A source that always sends the same message gives no surprise at all. A source where every choice is equally likely gives the most surprise it can.

Rare outcomes are surprising and common ones are not. The surprise of an outcome with probability $p$ is $\log_2 \frac{1}{p}$ , which gets bigger as $p$ gets smaller. The formula then averages those surprises, weighting each one by how often it happens. The minus sign is there because $\log_2 p$ is negative for $p$ below 1, and surprise should be positive.

A fair coin has two outcomes that are equally likely, so its average surprise is exactly 1 bit: one yes-or-no answer. A biased coin that lands heads 90% of the time is less surprising on average, so its entropy is lower.

Entropy also sets a limit for compression. It is the lowest average number of bits per symbol that any lossless code can reach in the long run. Data with a strong pattern has low entropy, so it can be packed into fewer bits. Claude Shannon introduced the idea in his 1948 paper “A Mathematical Theory of Communication”.

An everyday picture

Think of a weather report for a desert where it is sunny every single day. Saying “sunny” again tells you nothing new, so the report can be made very short. Now compare a place where the weather flips between sun, rain, snow and fog. Each report carries real news, so it needs more space. Entropy is the average amount of news per report.

Worked example: Entropy of a fair coin, a biased coin and a die

Three sources, each described by its outcome probabilities. We work out the entropy of each one in bits.

Fair coin: $p = (0.5,\ 0.5)$ . Each term is $0.5 \cdot \log_2 0.5 = 0.5 \cdot (-1) = -0.5$ . Adding the two terms gives $-1$ , and flipping the sign gives $H = 1$ bit.
Biased coin: $p = (0.9,\ 0.1)$ . The terms are $0.9 \cdot \log_2 0.9 \approx 0.9 \cdot (-0.152) = -0.137$ and $0.1 \cdot \log_2 0.1 \approx 0.1 \cdot (-3.322) = -0.332$ . Their sum is $-0.469$ , so $H = 0.469$ bits.
Fair six-sided die: six outcomes, each with $p = \frac{1}{6}$ . The six equal terms add up to $6 \cdot \frac{1}{6} \log_2 \frac{1}{6} = -\log_2 6$ , so $H = \log_2 6 \approx 2.585$ bits.
Biased coin over 1000 flips: $1000 \cdot 0.469 = 469$ bits, instead of 1000 bits for a fair coin.

Answer: A fair coin carries 1 bit of surprise per flip. The biased coin carries only 0.469 bits, because it is usually heads. A fair die carries 2.585 bits, a little more than 2, because six equal outcomes need a little more than two yes-or-no questions. A good code can average about 0.469 bits per flip for the biased coin, which is why predictable data compresses so well.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Leaving out the minus sign. Each $\log_2 p$ is zero or negative, so without the minus the answer would come out negative, and entropy is never negative.
Using the natural logarithm $\ln$ and calling the result bits. The answer is in bits only when you use $\log_2$ .
Using probabilities that do not add up to 1. The formula assumes the probabilities cover every possible outcome.
Assuming that the number of outcomes alone sets the entropy. A loaded die with six faces can have lower entropy than a fair die, because some faces come up more often than others.

Where you will meet it

Data compression tools such as ZIP and PNG, which store predictable data in fewer bits.
Measuring how predictable text is, for example how well a language model guesses the next letter or word.
Decision trees in machine learning, where a split is chosen to reduce entropy as much as possible.
Cryptography, where a secret key needs high entropy so that an attacker cannot guess it easily.

Questions people ask

What does "bits" mean here?

A bit is the information in one fair yes-or-no question. A fair coin flip has entropy 1 bit, because one answer, heads or tails, settles it.

Why is the entropy of a sure thing zero?

If one outcome has probability 1, every other probability is 0, and only the term for $p = 1$ is left. Since $\log_2 1 = 0$ , the entropy is 0. A message that always says the same thing tells you nothing new.

Who invented Shannon entropy?

Claude Shannon introduced it in his 1948 paper “A Mathematical Theory of Communication.” That paper founded the field of information theory.

Cross-Entropy Loss: Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.
Softmax: Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%.
Binary Number Value: Binary uses only the digits 0 and 1, and this formula turns a row of those digits into the ordinary number they stand for.