Sigmoid Function

Q: Where is the sigmoid steepest?

Right in the middle, at $z = 0$. That is where a small change in the input moves the output the most. Far out on either side, the curve is almost flat.

In one sentence: The sigmoid takes any number, however big or small, and squeezes it into a value between 0 and 1 that can be read as a chance.

$\sigma(z) = \frac{1}{1 + e^{-z}}$

Read it out loud: “Sigma of z equals one divided by the quantity one plus e to the power of negative z.”

What each symbol means

Symbol	Meaning
$\sigma(z)$	The sigmoid of $z$ : the output. It is always strictly between 0 and 1.
$z$	The input number, a raw score. It can be any real number: positive, negative or zero. It has no unit in most models.
$e$	Euler’s number, about 2.718. Here it is raised to the power $-z$ .

What the formula is doing

Many models need an output they can read as a probability, and a probability has to sit between 0 and 1. A raw score, however, can be any number at all, such as $-40$ or $+300$ . The sigmoid is the rule that bridges the two: whatever number goes in, a value between 0 and 1 comes out.

Here is what happens step by step. The term $e^{-z}$ is the same as $1 / e^{z}$ . When $z$ is a large positive number, $e^{-z}$ is almost 0, so the fraction is almost $1 / 1 = 1$ . When $z$ is a large negative number, $e^{-z}$ is huge, so the fraction is almost 0. When $z$ is exactly 0, $e^{0} = 1$ and the answer is $1/2$ .

The curve has an S shape. It climbs smoothly from near 0 on the left to near 1 on the right, and it passes through 0.5 when $z = 0$ . Inputs near the middle change the output the most. Very large inputs in either direction land close to 0 or 1 and barely move after that.

The sigmoid is also a smooth curve with no sharp corners. Training methods rely on that smoothness, because they measure how a small change in $z$ changes the output. That is why the sigmoid appears in logistic regression and in early neural networks.

An everyday picture

Think of a dimmer switch with a stop at each end. Turn it all the way one way and the light is fully off; turn it all the way the other way and it is fully on. Around the middle, a small turn changes the brightness a lot, while near the ends you can turn much further without any visible change. The sigmoid turns a raw input into a brightness somewhere between off (0) and fully on (1).

Worked example: Three inputs: positive, zero and negative

Apply the sigmoid to three raw scores: $z = 2$ , $z = 0$ and $z = -2$ . All values are unitless here.

For $z = 2$ : $\sigma(2) = \frac{1}{1 + e^{-2}} = \frac{1}{1 + 0.1353} \approx 0.881$ .
For $z = 0$ : $\sigma(0) = \frac{1}{1 + e^{0}} = \frac{1}{1 + 1} = 0.5$ .
For $z = -2$ : $\sigma(-2) = \frac{1}{1 + e^{2}} = \frac{1}{1 + 7.389} \approx 0.119$ .

Answer: A raw score of $+2$ becomes about 0.881, a score of 0 lands exactly on 0.5, and a score of $-2$ drops to about 0.119. Positive scores lean toward 1, negative scores lean toward 0, and 0 sits in the middle.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Mixing up $e^{-z}$ with $e^{z}$ . The minus sign decides whether big inputs push the output toward 1 or toward 0. With $z = 2$ the answer must be above 0.5.
Expecting the output to be exactly 0 or exactly 1. The sigmoid gets close to 0 or 1 but never reaches them for any finite input.
Treating every output between 0 and 1 as a well-calibrated probability. A model can output 0.9 for answers that turn out right only half the time.
Using the sigmoid for a whole list of scores that must add up to 1. The sigmoid works on one score at a time; for a list, use softmax.

Where you will meet it

Logistic regression, the classic model for yes-or-no questions, which passes a weighted sum through a sigmoid.
The output layer of a binary neural network classifier, such as a spam-or-not-spam score.
The gates inside LSTM recurrent networks, which use sigmoids to decide how much information to keep or forget.
Early neural networks, before ReLU became the usual choice in deep networks.

Questions people ask

Why is it called sigmoid?

Sigmoid means S-shaped. Plot the curve and it looks like a stretched letter S.

How is the sigmoid different from softmax?

The sigmoid works on one score and gives one number between 0 and 1. Softmax works on a whole list and makes the list add up to 1. See the softmax page for that rule.

Can the sigmoid output be negative?

No. Because $e^{-z}$ is always positive, the bottom of the fraction is always bigger than 1, so the result stays strictly between 0 and 1.

Where is the sigmoid steepest?

Right in the middle, at $z = 0$ . That is where a small change in the input moves the output the most. Far out on either side, the curve is almost flat.

Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds them up with a bias, and then decides how strongly to fire.
Softmax: Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%.
Cross-Entropy Loss: Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.