Softmax

Q: What is the difference between softmax and sigmoid?

Sigmoid squeezes one score into a number between 0 and 1. Softmax handles a whole list and makes the results add up to 1. With exactly two answers, softmax gives the same result as a sigmoid of the difference between the two scores.

In one sentence: Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%.

$\operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$

Read it out loud: “The softmax of score number i equals e to the power of that score, divided by the sum of e to the power of every score.”

What each symbol means

Symbol	Meaning
$z$	The list of raw scores (also called logits). A model produces one score per possible answer. They can be any real numbers, even negative.
$z_i$	One score from the list, the one for answer number $i$ .
$K$	How many scores are in the list, so how many possible answers there are.
$e$	Euler’s number, about 2.718. Raising $e$ to a power makes every number positive.
$\sum_{j=1}^{K}$	“Add up for every $j$ from 1 to $K$ ”: go through the whole list and total it.
$\operatorname{softmax}(z)_i$	The result for answer $i$ : a number between 0 and 1, read as the probability of that answer.

What the formula is doing

A classifier, for example a model that looks at a photo and decides between cat, dog and bird, first produces one raw score per answer. The scores are not probabilities yet: they can be negative, they can be huge, and they do not add up to anything in particular.

Softmax fixes that in two moves. First it raises $e$ to the power of every score. That turns every number positive, and it stretches the gaps: a score that is a bit bigger becomes a lot bigger after $e^{z}$ .

Second, it divides each of those positive numbers by their total. Dividing by the total is what makes the results add up to exactly 1, the same way you turn counts into percentages by dividing by the overall count.

The order never changes: the biggest score always gets the biggest probability. Only the sizes change. Adding the same number to every score changes nothing at all, because that factor appears on the top and the bottom and cancels out.

An everyday picture

Think of a talent show where judges hand out raw points. Softmax is the host who converts those points into each act’s share of the audience vote. Everybody gets a slice, the slices fill the whole pie, and the act with the most points gets the biggest slice, a little bigger than the raw points alone would suggest.

Worked example: A photo classifier choosing between cat, dog and bird

The model gives the scores $z = (2.0,\ 1.0,\ 0.1)$ for cat, dog and bird.

Raise $e$ to each score: $e^{2.0} \approx 7.389$ , $e^{1.0} \approx 2.718$ , $e^{0.1} \approx 1.105$ .
Add them up: $7.389 + 2.718 + 1.105 = 11.213$ .
Divide each one by the total: $\frac{7.389}{11.213} \approx 0.659, \quad \frac{2.718}{11.213} \approx 0.242, \quad \frac{1.105}{11.213} \approx 0.099$
Check: $0.659 + 0.242 + 0.099 = 1.000$ .

Answer: The model says cat with about 65.9% confidence, dog 24.2% and bird 9.9%. The cat score was only twice the dog score, but its probability is almost three times bigger: that is the stretching effect of $e^{z}$ .

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Dividing each score by the plain sum of scores. That breaks with negative scores and is not softmax; the $e^{z}$ step comes first.
Computing $e^{z}$ directly for big scores like 1000. The number overflows. Real code subtracts the largest score from every score first, which gives the same answer (see the last paragraph above) without the overflow.
Reading the output as the model’s true certainty. Softmax always produces confident-looking percentages, even for a photo of something the model has never seen.
Applying softmax twice, for example inside the model and again inside the loss function. Many libraries’ cross-entropy functions already include softmax.

Where you will meet it

The last layer of almost every image, text and speech classifier, to turn scores into class probabilities.
Large language models, which use it to turn a score for every word in the vocabulary into the probability of the next word.
The attention mechanism in transformers, where softmax decides how much each word should look at every other word.
Reinforcement learning, where a softmax over action values picks actions at random but favors the better ones.

Questions people ask

Why use e and not 2 or 10?

Any base above 1 would turn scores into positive numbers and keep their order. Base $e$ is used because its derivative is itself, which makes the math of training (gradients) short and clean.

What is the difference between softmax and sigmoid?

Sigmoid squeezes one score into a number between 0 and 1. Softmax handles a whole list and makes the results add up to 1. With exactly two answers, softmax gives the same result as a sigmoid of the difference between the two scores.

What is temperature in softmax?

Dividing every score by a number $T$ before softmax. $T$ above 1 flattens the percentages (more random choices), $T$ below 1 sharpens them toward the top answer. Chatbots expose this as the temperature setting.

Why is it called softmax?

A hard max would put 100% on the biggest score and 0% on the rest. Softmax does a soft version of that: the biggest score wins most, but every answer keeps a share.

Sigmoid Function: The sigmoid takes any number, however big or small, and squeezes it into a value between 0 and 1 that can be read as a chance.
Cross-Entropy Loss: Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.
Scaled Dot-Product Attention: Attention lets a model look at every item in a list and decide how much each one matters for the item it is working on.

Sources

Published 2026-10-22. Plain-text copy for AI agents and readers: ai-ml/softmax.md.