# Shannon Entropy > Shannon entropy is one number, in bits, that says how unpredictable a source is. The more predictable it is, the less space its messages need. Page: https://formulabro.yunusemrevurgun.com/computer-science/shannon-entropy/ Category: Computer Science Published: 2026-10-22 Updated: 2026-10-22 ## Formula $$ H = -\sum_{i=1}^{n} p_i \log_2 p_i $$ Read it out loud: "H equals minus the sum, for every outcome i from 1 to n, of p sub i times the base-2 logarithm of p sub i. The answer is measured in bits." ## What each symbol means - $H$: The entropy of the source, in **bits**. One bit is the information in one fair yes-or-no answer. - $n$: How many different possible outcomes there are, such as 2 for a coin or 6 for a die. - $i$: Which outcome we are looking at, counted from 1 to $n$. - $p_i$: The probability of outcome $i$: a number from 0 to 1. All the $p_i$ together add up to 1. - $\log_2 p_i$: The base-2 logarithm of the probability. It is zero or negative for any probability at most 1, which is why the formula has a minus sign in front. - $-\sum_{i=1}^{n}$: "Add up the terms for every outcome, then flip the sign." Flipping the sign makes the final answer zero or positive. ## What the formula is doing Shannon entropy answers one simple question: on average, how surprised should you be by the next message from a source? A source that always sends the same message gives no surprise at all. A source where every choice is equally likely gives the most surprise it can. Rare outcomes are surprising and common ones are not. The surprise of an outcome with probability $p$ is $\log_2 \frac{1}{p}$, which gets bigger as $p$ gets smaller. The formula then averages those surprises, weighting each one by how often it happens. The minus sign is there because $\log_2 p$ is negative for $p$ below 1, and surprise should be positive. A fair coin has two outcomes that are equally likely, so its average surprise is exactly 1 bit: one yes-or-no answer. A biased coin that lands heads 90% of the time is less surprising on average, so its entropy is lower. Entropy also sets a limit for compression. It is the lowest average number of bits per symbol that any lossless code can reach in the long run. Data with a strong pattern has low entropy, so it can be packed into fewer bits. Claude Shannon introduced the idea in his 1948 paper "A Mathematical Theory of Communication". ## An everyday picture Think of a weather report for a desert where it is sunny every single day. Saying "sunny" again tells you nothing new, so the report can be made very short. Now compare a place where the weather flips between sun, rain, snow and fog. Each report carries real news, so it needs more space. Entropy is the average amount of news per report. ## Worked example: Entropy of a fair coin, a biased coin and a die Three sources, each described by its outcome probabilities. We work out the entropy of each one in bits. 1. Fair coin: $p = (0.5,\ 0.5)$. Each term is $0.5 \cdot \log_2 0.5 = 0.5 \cdot (-1) = -0.5$. Adding the two terms gives $-1$, and flipping the sign gives $H = 1$ bit. 2. Biased coin: $p = (0.9,\ 0.1)$. The terms are $0.9 \cdot \log_2 0.9 \approx 0.9 \cdot (-0.152) = -0.137$ and $0.1 \cdot \log_2 0.1 \approx 0.1 \cdot (-3.322) = -0.332$. Their sum is $-0.469$, so $H = 0.469$ bits. 3. Fair six-sided die: six outcomes, each with $p = \frac{1}{6}$. The six equal terms add up to $6 \cdot \frac{1}{6} \log_2 \frac{1}{6} = -\log_2 6$, so $H = \log_2 6 \approx 2.585$ bits. 4. Biased coin over 1000 flips: $1000 \cdot 0.469 = 469$ bits, instead of 1000 bits for a fair coin. **Answer:** A fair coin carries **1 bit** of surprise per flip. The biased coin carries only **0.469 bits**, because it is usually heads. A fair die carries **2.585 bits**, a little more than 2, because six equal outcomes need a little more than two yes-or-no questions. A good code can average about 0.469 bits per flip for the biased coin, which is why predictable data compresses so well. ## Common mistakes - Leaving out the minus sign. Each $\log_2 p$ is zero or negative, so without the minus the answer would come out negative, and entropy is never negative. - Using the natural logarithm $\ln$ and calling the result bits. The answer is in bits only when you use $\log_2$. - Using probabilities that do not add up to 1. The formula assumes the probabilities cover every possible outcome. - Assuming that the number of outcomes alone sets the entropy. A loaded die with six faces can have lower entropy than a fair die, because some faces come up more often than others. ## Where you will meet it - Data compression tools such as ZIP and PNG, which store predictable data in fewer bits. - Measuring how predictable text is, for example how well a language model guesses the next letter or word. - Decision trees in machine learning, where a split is chosen to reduce entropy as much as possible. - Cryptography, where a secret key needs high entropy so that an attacker cannot guess it easily. ## Questions people ask ### What does "bits" mean here? A bit is the information in one fair yes-or-no question. A fair coin flip has entropy 1 bit, because one answer, heads or tails, settles it. ### Why is the entropy of a sure thing zero? If one outcome has probability 1, every other probability is 0, and only the term for $p = 1$ is left. Since $\log_2 1 = 0$, the entropy is 0. A message that always says the same thing tells you nothing new. ### Who invented Shannon entropy? Claude Shannon introduced it in his 1948 paper "A Mathematical Theory of Communication." That paper founded the field of information theory. ## Related formulas - [Cross-Entropy Loss](https://formulabro.yunusemrevurgun.com/ai-ml/cross-entropy-loss/) - [Softmax](https://formulabro.yunusemrevurgun.com/ai-ml/softmax/) - [Binary Number Value](https://formulabro.yunusemrevurgun.com/computer-science/binary-number-value/) ## Sources - [Entropy (information theory), Wikipedia](https://en.wikipedia.org/wiki/Entropy_(information_theory)) - [A Mathematical Theory of Communication, Wikipedia](https://en.wikipedia.org/wiki/A_Mathematical_Theory_of_Communication) From Formula Bro (https://formulabro.yunusemrevurgun.com/), a side project developed by yunusemrevurgun.com.