# Softmax > Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%. Page: https://formulabro.yunusemrevurgun.com/ai-ml/softmax/ Category: AI & Machine Learning Published: 2026-10-22 Updated: 2026-10-22 ## Formula $$ \operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} $$ Read it out loud: "The softmax of score number i equals e to the power of that score, divided by the sum of e to the power of every score." ## What each symbol means - $z$: The list of raw scores (also called **logits**). A model produces one score per possible answer. They can be any real numbers, even negative. - $z_i$: One score from the list, the one for answer number $i$. - $K$: How many scores are in the list, so how many possible answers there are. - $e$: Euler's number, about 2.718. Raising $e$ to a power makes every number positive. - $\sum_{j=1}^{K}$: "Add up for every $j$ from 1 to $K$": go through the whole list and total it. - $\operatorname{softmax}(z)_i$: The result for answer $i$: a number between 0 and 1, read as the probability of that answer. ## What the formula is doing A classifier, for example a model that looks at a photo and decides between *cat*, *dog* and *bird*, first produces one raw score per answer. The scores are not probabilities yet: they can be negative, they can be huge, and they do not add up to anything in particular. Softmax fixes that in two moves. First it raises $e$ to the power of every score. That turns every number positive, and it stretches the gaps: a score that is a bit bigger becomes a lot bigger after $e^{z}$. Second, it divides each of those positive numbers by their total. Dividing by the total is what makes the results add up to exactly 1, the same way you turn counts into percentages by dividing by the overall count. The order never changes: the biggest score always gets the biggest probability. Only the sizes change. Adding the same number to every score changes nothing at all, because that factor appears on the top and the bottom and cancels out. ## An everyday picture Think of a talent show where judges hand out raw points. Softmax is the host who converts those points into each act's share of the audience vote. Everybody gets a slice, the slices fill the whole pie, and the act with the most points gets the biggest slice, a little bigger than the raw points alone would suggest. ## Worked example: A photo classifier choosing between cat, dog and bird The model gives the scores $z = (2.0,\ 1.0,\ 0.1)$ for *cat*, *dog* and *bird*. 1. Raise $e$ to each score: $e^{2.0} \approx 7.389$, $e^{1.0} \approx 2.718$, $e^{0.1} \approx 1.105$. 2. Add them up: $7.389 + 2.718 + 1.105 = 11.213$. 3. Divide each one by the total: $$\frac{7.389}{11.213} \approx 0.659, \quad \frac{2.718}{11.213} \approx 0.242, \quad \frac{1.105}{11.213} \approx 0.099$$ 4. Check: $0.659 + 0.242 + 0.099 = 1.000$. **Answer:** The model says *cat* with about **65.9%** confidence, *dog* 24.2% and *bird* 9.9%. The cat score was only twice the dog score, but its probability is almost three times bigger: that is the stretching effect of $e^{z}$. ## Common mistakes - Dividing each score by the plain sum of scores. That breaks with negative scores and is not softmax; the $e^{z}$ step comes first. - Computing $e^{z}$ directly for big scores like 1000. The number overflows. Real code subtracts the largest score from every score first, which gives the same answer (see the last paragraph above) without the overflow. - Reading the output as the model's true certainty. Softmax always produces confident-looking percentages, even for a photo of something the model has never seen. - Applying softmax twice, for example inside the model and again inside the loss function. Many libraries' cross-entropy functions already include softmax. ## Where you will meet it - The last layer of almost every image, text and speech classifier, to turn scores into class probabilities. - Large language models, which use it to turn a score for every word in the vocabulary into the probability of the next word. - The attention mechanism in transformers, where softmax decides how much each word should look at every other word. - Reinforcement learning, where a softmax over action values picks actions at random but favors the better ones. ## Questions people ask ### Why use e and not 2 or 10? Any base above 1 would turn scores into positive numbers and keep their order. Base $e$ is used because its derivative is itself, which makes the math of training (gradients) short and clean. ### What is the difference between softmax and sigmoid? Sigmoid squeezes **one** score into a number between 0 and 1. Softmax handles a **whole list** and makes the results add up to 1. With exactly two answers, softmax gives the same result as a sigmoid of the difference between the two scores. ### What is temperature in softmax? Dividing every score by a number $T$ before softmax. $T$ above 1 flattens the percentages (more random choices), $T$ below 1 sharpens them toward the top answer. Chatbots expose this as the *temperature* setting. ### Why is it called softmax? A hard max would put 100% on the biggest score and 0% on the rest. Softmax does a soft version of that: the biggest score wins most, but every answer keeps a share. ## Related formulas - [Sigmoid Function](https://formulabro.yunusemrevurgun.com/ai-ml/sigmoid-function/) - [Cross-Entropy Loss](https://formulabro.yunusemrevurgun.com/ai-ml/cross-entropy-loss/) - [Scaled Dot-Product Attention](https://formulabro.yunusemrevurgun.com/ai-ml/scaled-dot-product-attention/) ## Sources - [Softmax function, Wikipedia](https://en.wikipedia.org/wiki/Softmax_function) - [Goodfellow, Bengio and Courville, Deep Learning, chapter 6.2.2.3 (Softmax units)](https://www.deeplearningbook.org/contents/mlp.html) From Formula Bro (https://formulabro.yunusemrevurgun.com/), a side project developed by yunusemrevurgun.com.