Softmax Temperature

Q: Which temperature is normal?

$T = 1$. With that value the formula is the ordinary softmax, and the scores are not changed.

In one sentence: Temperature is a dial that makes a model’s choices more focused when it is low and more varied when it is high.

$p_i = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}}$

Read it out loud: “The probability p sub i, for answer number i, equals e raised to the power z sub i divided by T, all divided by the sum, over every answer j, of e raised to the power z sub j divided by T. The letter e is about 2.718. The letter z is a raw score, one number the model gives each answer. The letter T is the temperature, a positive number. The capital sigma means add up, and K is the number of answers.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Softmax, the rule that makes probabilities: A softmax turns a list of raw scores into probabilities. Each output is a number between 0 and 1, and all the outputs add up to exactly 1. The larger a score is, the larger its probability. Softmax is used at the end of a model that must choose one answer out of several, such as the next word in a sentence.
The number $e$ and exponentials, $e^{x}$: The number $e$ is a fixed number, about 2.71828. Writing $e^{x}$ means $e$ raised to the power $x$ , which is $e$ multiplied by itself $x$ times. So $e^{0} = 1$ , $e^{1} \approx 2.718$ , and $e^{2} \approx 7.389$ . An exponential is always positive. A negative power gives a small positive number, so $e^{-1} \approx 0.368$ , and bigger powers give much bigger numbers.
Raw scores, also called logits: A raw score, also called a logit, is the number a model gives each possible answer before softmax turns the scores into probabilities. It can be any number, even a negative one, and it has no unit. The scores in the example are 2.0, 1.0 and 0.1. They are not probabilities, because they do not add up to 1.
Probability and percent: A probability is a number from 0 to 1 that says how likely something is. 0 means impossible, 1 means certain, and 0.5 means a 50-50 chance. Multiplying by 100 gives a percent, so 0.659 is 65.9 percent. The probabilities for all the answers always add up to 1.
The sum sign, subscripts and the letters $i$ , $j$ , $K$: The capital sigma, $\sum$ , means add up. In $\sum_{j=1}^{K}$ the letter $j$ runs from 1 to $K$ , so the sum covers every answer. The letter $K$ is the number of answers. A subscript such as $z_i$ names one score in the list, so $z_1$ is the first answer’s score, and $z_i$ is the score of whichever answer $i$ we are working on.
Fractions and dividing by $T$: A fraction bar means divide: $\frac{a}{b}$ is $a$ divided by $b$ . A slash does the same job, so $z_i / T$ means the score $z_i$ divided by the temperature $T$ . Writing $e^{z_i / T}$ means first divide the score by $T$ , then raise $e$ to that power. The big fraction in the whole formula means divide the top part by the total on the bottom.

What each symbol means

Symbol	Meaning
$p_i$	The probability of answer number $i$ , after the temperature is applied. It is a fraction between 0 and 1 with no unit, and the probabilities of all the answers add up to exactly 1. In the example at $T = 0.5$ , the probability of cat is 0.864, which is 86.4 percent.
$z_i$	The raw score, or logit, of answer number $i$ , the number the model gave it before the temperature is applied. It has no unit and can be any real number, even a negative one. In the example, the raw score of cat is $z = 2.0$ .
$i$	The position of the answer being worked out, a counting number from 1 to $K$ . It has no unit. It picks out one answer from the list, such as cat when $i = 1$ .
$T$	The temperature, a positive number with no unit. $T = 1$ gives the usual softmax with no change to the scores. A smaller $T$ makes the top answer stand out more, and a larger $T$ spreads the probability out. $T$ must never be zero, because dividing by zero is not allowed.
$j$	The running number over all the answers, counting from 1 to $K$ . It is used inside the sum on the bottom of the fraction. It has no unit.
$K$	The number of answers, a count of how many scores are in the list. In the example there are three answers, so $K = 3$ . It has no unit.
$z_j$	The raw score of answer number $j$ . It is used on the bottom of the fraction, so that all the scaled scores are added up into one total. Like $z_i$ , it has no unit.
$e$	Euler’s number, about 2.718, the base of natural exponentials. Raising $e$ to any power gives a positive number, which is why every softmax output is positive. It has no unit.
$\sum_{j=1}^{K}$	The add-up sign over every answer $j$ from 1 to $K$ . It gives one total: the sum of all the exponentials $e^{z_j/T}$ , which becomes the bottom of each fraction.

The formula piece by piece

$z_i$: Start with one raw score. $z_i$ is the score the model gave answer $i$ . The bigger the score, the more the model prefers that answer. Scores can be any number, positive or negative.
$\frac{z_i}{T}$: Divide the score by the temperature. This is the only new step. Dividing by a number smaller than 1 makes the score bigger, and dividing by a number bigger than 1 makes it smaller. For example, $2.0 / 0.5 = 4.0$ , and $2.0 / 2 = 1.0$ .
$e^{z_i / T}$: Raise $e$ to the power of that scaled score. This makes the value positive. A bigger scaled score gives a much bigger number, because each extra unit of score multiplies the result by about 2.718.
$\sum_{j=1}^{K} e^{z_j / T}$: Add up the exponentials of all the answers. This total is the same for every answer, so it is the bottom of each fraction. Dividing by it is what makes all the probabilities add up to 1.
$p_i = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}}$: The probability of answer $i$ equals $e$ raised to its scaled score, divided by the total of $e$ raised to every scaled score. Changing $T$ changes the scaled scores, and so it changes how spread out the probabilities are.
$p_i = \frac{\exp(z_i / T)}{\sum_{j=1}^{K} \exp(z_j / T)}$: The same formula written with $\exp(x)$ , which means $e^{x}$ . Many programs and papers use this notation, so you may see it in place of the $e$ form.

What the formula is doing, and why

The input is a list of raw scores, one for each answer, plus one temperature $T$ . The output is a list of probabilities, one for each answer, that add up to 1. Softmax works in three steps: divide each score by $T$ , raise $e$ to each result, then divide each result by the total. Temperature changes only the first step.

Why divide by $T$ ? A small $T$ makes the scores farther apart after they are divided. Dividing by $0.5$ doubles every score, so the gaps between the scores double. Those bigger gaps push the top answer further ahead after the exponentials, so it takes more of the probability. As $T$ gets closer to zero, the top answer takes almost all of it.

Why exponentials, and why $e$ ? The exponential makes every value positive, which a probability needs. It also has a useful property: when two exponentials are divided, the result is $e$ raised to the difference of their powers. So a gap between two scores becomes a ratio between two probabilities. This is one reason softmax is built on $e$ .

Why a large $T$ spreads things out. Dividing by $2$ halves every score, so the gaps shrink. The exponentials then move closer together, and the probabilities move toward equal shares. Lower-ranked answers get picked more often, so the output becomes more varied.

The ranking never changes. Any positive $T$ divides every score by the same number, so the order of the scores stays the same. The best answer always stays on top and the worst at the bottom. Temperature only changes how much probability the top answer keeps. $T = 0$ is not allowed, because dividing by zero is undefined. Code that always wants the top answer uses an argmax instead, which means the position of the largest value.

An everyday picture

Think of a spotlight on a stage. At a low temperature the beam is narrow, so only the brightest performer is lit. At a high temperature the beam spreads out, and every performer on stage gets some light. The performers and their order stay the same. Only the amount of light changes.

Worked example: Choosing the next word at two temperatures

A language model scores three possible next words, cat, dog and bird, with raw scores $z = (2.0,\ 1.0,\ 0.1)$ . At the usual $T = 1$ , the probability of cat is 65.9 percent. Now we use $T = 0.5$ and then $T = 2$ to see how the spread changes.

Start at the usual setting, $T = 1$ . Dividing by 1 changes nothing, so the scores are used as they are. We raise $e$ to each score: $e^{2.0} \approx 7.389$ , $e^{1.0} \approx 2.718$ and $e^{0.1} \approx 1.105$ . Their total is about 11.213, and the share for cat is $\frac{7.389}{11.213} \approx 0.659$ , or 65.9 percent. This is the baseline to compare against.
Divide each score by $T = 0.5$ . Dividing by one half is the same as multiplying by 2, so every score doubles: $z / T = (4.0,\ 2.0,\ 0.2)$ . The gaps between the scores get bigger, and that is what makes the output sharper.
Raise $e$ to each new score and add them up. $e^{4.0} \approx 54.598$ , $e^{2.0} \approx 7.389$ and $e^{0.2} \approx 1.221$ . The total is about 63.209. The last digit can differ from adding the rounded parts by hand, because each value has been rounded.
Divide each result by the total. For cat, $\frac{54.598}{63.209} \approx 0.864$ . For dog, $\frac{7.389}{63.209} \approx 0.117$ . For bird, $\frac{1.221}{63.209} \approx 0.019$ . Adding them gives $0.864 + 0.117 + 0.019 = 1.000$ , which confirms that the probabilities add up to 1.
Repeat with $T = 2$ . Dividing by 2 halves every score: $z / T = (1.0,\ 0.5,\ 0.05)$ . Raising $e$ to these gives about $2.718$ , $1.649$ and $1.051$ , with a total of $5.418$ . Dividing gives cat $\approx 0.502$ , dog $\approx 0.304$ and bird $\approx 0.194$ , which again add to 1.
Compare the three settings. Cat goes from 65.9 percent at $T = 1$ , up to 86.4 percent at $T = 0.5$ , and down to 50.2 percent at $T = 2$ . Bird goes from about 1.9 percent at $T = 0.5$ up to 19.4 percent at $T = 2$ . The order never changes, so cat stays on top in all three settings.

Answer: At $T = 0.5$ , cat rises to about 86.4 percent and bird falls to 1.9 percent. At $T = 2$ , cat drops to about 50.2 percent and bird rises to 19.4 percent. Cat is the top answer at every temperature. Only the spread changes.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Setting $T = 0$ and dividing by it. The temperature must be greater than zero. Code that only wants the top answer uses an argmax instead, which is the position of the largest score.
Expecting a high temperature to keep the top answer dominant. A higher $T$ spreads the probability out, so the top answer loses its lead, even though it stays on top.
Applying the temperature to the finished probabilities. The division has to happen on the raw scores, before the $e$ step. Dividing the probabilities by $T$ does not give the same result.
Thinking a temperature setting makes a model smarter or more accurate. It changes only how the same scores are turned into probabilities. The scores themselves stay the same.

Where you will meet it

Sampling text from large language models, where a temperature setting controls how predictable or varied the chosen words are.
Knowledge distillation, where a large teacher model’s scores are softened with a high temperature, so that a smaller student model can learn from how the wrong answers rank.
Temperature scaling, a method that adjusts how confident a trained classifier’s probabilities look, without changing its ranking of the answers.
Exploration in reinforcement learning, where a program learns by trying actions and collecting rewards. A softmax with a temperature picks the better actions more often without always picking the single best one.

Questions people ask

What is the temperature for, in one sentence?

It divides every raw score by $T$ before softmax, so a small $T$ makes the top answer stand out, and a large $T$ makes the choices more even.

Which temperature is normal?

$T = 1$ . With that value the formula is the ordinary softmax, and the scores are not changed.

Can temperature change which answer wins?

No. Dividing every score by the same positive number keeps their order. Only the spread of the probabilities changes.

What happens as $T$ gets very close to zero?

The biggest score takes almost all the probability, so the output comes close to always picking the top answer.

Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.
Cross-Entropy Loss: Cross-entropy loss gives a small penalty when the model gave the right answer a high probability, and a big penalty when it gave the right answer a small probability.
Perplexity: Perplexity counts how many words a language model is torn between, on average, when it guesses the next word. Lower is better.