Logistic Regression

Q: How is logistic regression different from linear regression?

Linear regression predicts a number, such as a price. Logistic regression predicts a probability for a class, and the sigmoid keeps the output between 0 and 1.

Q: Why is it called logistic?

The sigmoid curve it uses is also called the logistic function, and that is where the name comes from.

Q: Can the output ever be exactly 0 or 1?

Not in exact arithmetic. The sigmoid gets very close to 0 for large negative scores and very close to 1 for large positive scores, but it never reaches either end.

Q: Can logistic regression handle more than two classes?

Yes. The method is extended to several classes, usually by using softmax in place of the sigmoid.

In one sentence: Logistic regression takes the same weighted sum as a straight line, then squeezes it through an S-shaped curve so the result is a probability between 0 and 1 for a yes-or-no question.

$p = \sigma(w x + b) = \frac{1}{1 + e^{-(w x + b)}}$

Read it out loud: “p equals sigma of w x plus b, which equals one over one plus e to the power of minus the quantity w x plus b. Here p is the probability of class 1, sigma is the sigmoid curve, w is the weight, x is the input, b is the bias, and e is Euler's number, about 2.718.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Probability between 0 and 1, and percent: A probability is a number from 0 to 1 that says how likely an outcome is. 0 means it cannot happen, 1 means it is certain, and 0.5 means an even chance. Multiply by 100 to read it as a percentage, so 0.731 is 73.1 percent. A logistic regression model gives a probability for each new case, and the answer is never more than 1 or less than 0.
The sigmoid, and the Greek letter $\sigma$ (sigma): The Greek letter $\sigma$ , said SIG-muh, is used here as the name of a rule, the sigmoid. It takes any number, positive, negative or zero, and returns a value between 0 and 1. A graph of it looks like a stretched letter S, which is where the name comes from. Writing $\sigma(z)$ means ‘apply the sigmoid to the number $z$ ’.
Euler’s number $e$ and negative exponents: The letter $e$ is a fixed number, about 2.71828, called Euler’s number. $e^{x}$ means $e$ raised to the power $x$ , so $e^{1} \approx 2.718$ and $e^{2} \approx 7.389$ . A minus sign in the exponent means divide: $e^{-x} = \frac{1}{e^{x}}$ . So $e^{-1} = \frac{1}{2.718} \approx 0.368$ , which is a small positive number.
Odds and log-odds: Odds compare the chance of an outcome with the chance it does not happen. If the probability of passing is 0.75, the odds are 3 to 1, because passing is three times as likely as failing. The log-odds, the raw score $z = w x + b$ in this model, is the logarithm of the odds. It can be any number, and the sigmoid turns it into a probability.
A threshold: A threshold is a cutoff number that turns a probability into a decision. With a threshold of 0.5, the model says ‘yes’ whenever the probability is 0.5 or more, and ‘no’ otherwise. The threshold is a choice made by the person using the model, and it can be set higher or lower depending on whether false alarms or misses cost more.

What each symbol means

Symbol	Meaning
$p$	The probability that the outcome is class 1, for example that a student passes. It is always between 0 and 1, and multiplying by 100 gives a percentage. It has no unit, because it is a share of all the cases.
$\sigma$	The sigmoid function, also called the logistic function. It takes any real number and returns a value strictly between 0 and 1. It is a fixed rule, not a number you learn, and it is what turns the raw score into a probability.
$w$	The weight, the number that says how strongly the input pushes the probability up or down. A positive weight means a larger $x$ gives a higher probability. In the example, $w = 0.8$ means each extra hour of study adds 0.8 to the raw score. Its unit is the unit of the score divided by the unit of $x$ .
$x$	The input, a number you measure for each case, such as hours of study. Its unit depends on what you measure, and in the example it is hours. The model uses $x$ together with $w$ and $b$ to produce the probability.
$b$	The bias, also called the intercept. It shifts the whole curve left or right, and it sets the input value where the probability crosses 0.5. Its unit is the same as the raw score, which has no everyday unit in most models. In the example, $b = -3$ .
$e$	Euler’s number, a fixed value of about 2.71828. It appears in the formula because the sigmoid is built from exponentials, and raising $e$ to a negative power produces the smooth S-shaped curve. It has no unit.

The formula piece by piece

$w x$: Start with the input scaled by the weight. Multiplying the input by $w$ says how strongly this input pushes the answer, so a bigger $x$ with a positive $w$ gives a bigger push toward class 1.
$w x + b$: Add the bias to the weighted input. The result is the raw score, also called the log-odds, which can be any number. It is exactly the same weighted sum that a straight-line model would use, and it is still not a probability.
$e^{-(w x + b)}$: Compute $e$ raised to the negative of the raw score. This number is always positive. It is close to 0 when the raw score is large and positive, and it is large when the raw score is large and negative. This is the step that makes the curve bend.
$\frac{1}{1 + e^{-(w x + b)}}$: Add 1 to the positive number and divide 1 by the total. The answer is always between 0 and 1: near 1 for a large positive raw score, near 0 for a large negative one, and exactly one half when the raw score is 0.
$p = \sigma(w x + b) = \frac{1}{1 + e^{-(w x + b)}}$: The whole formula in one sentence: the probability of class 1 equals the sigmoid of the weighted input plus the bias, which is one divided by one plus $e$ raised to the minus of that weighted sum.

What the formula is doing, and why

Linear regression predicts a number, but many questions have a yes-or-no answer: will the student pass, is the email spam, will the loan be repaid. For these you want a probability. Logistic regression starts from the same weighted sum as a straight line, $z = w x + b$ , and then passes it through the sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ . The value $z$ is the raw score, which is also called the log-odds, and it can be any size.

What does the sigmoid do? It does the squeezing. When $z$ is a large positive number, $e^{-z}$ is tiny and the output is close to 1. When $z$ is a large negative number, $e^{-z}$ is huge and the output is close to 0. When $z$ is exactly 0, the output is exactly 0.5. The curve rises smoothly between those extremes, so a small change in the input gives a small change in the probability.

How are $w$ and $b$ learned? The weights $w$ and $b$ are learned from labeled examples, where the true answer is known for each case. They are chosen to make the observed answers as likely as possible, a rule called maximum likelihood. For this kind of model that rule is the same as making the cross-entropy loss as small as possible. Once the model is trained, a yes-or-no answer comes from a threshold: predict class 1 when $p \ge 0.5$ and class 0 otherwise. Because $\sigma(0) = 0.5$ , that rule is the same as asking whether $w x + b \ge 0$ .

Why not always use 0.5? The threshold of 0.5 is a default, not a law. When a false alarm costs more than a miss, as in some medical screening, a different threshold is wiser. With one input, the boundary between the two classes is the single point where $z = 0$ , which is $x = -b / w$ .

An everyday picture

Think of a sliding scale that runs from very unlikely to very likely. The weighted sum is a raw score that can be any size, like points on a scoreboard. The sigmoid is the translator that turns any score into a position on the 0 to 100 percent scale. A score of zero lands exactly in the middle, at 50 percent.

Worked example: Will a student pass a test after studying?

A model uses hours studied $x$ . It has learned the weight $w = 0.8$ per hour and the bias $b = -3$ . We find the probability of passing for a student who studies 5 hours and for one who studies 2 hours. We use a threshold of 0.5. The probabilities are made up for this example to show the method.

Compute the raw score for 5 hours. The weighted sum is $z = 0.8 \times 5 - 3 = 4 - 3 = 1.0$ . Each hour adds 0.8 to the score, and the bias takes away 3, so this student has a raw score of 1.0.
Find $e^{-z}$ for this score. The formula needs $e^{-1.0}$ , which equals $\frac{1}{e^{1.0}} = \frac{1}{2.718} \approx 0.368$ . Because the raw score is positive, the number is less than 1, which means the probability will be above one half.
Apply the sigmoid. The probability is $p = \frac{1}{1 + e^{-1.0}} = \frac{1}{1 + 0.368} \approx 0.731$ . Multiplying by 100 gives 73.1 percent, the chance that this student passes.
Compare with the threshold. Since $0.731 \geq 0.5$ , the model predicts pass for this student. The threshold is the cutoff that turns the probability into a yes-or-no decision.
Repeat the steps for 2 hours. The raw score is $z = 0.8 \times 2 - 3 = 1.6 - 3 = -1.4$ . The student has studied less, so the raw score is negative.
Apply the sigmoid to the negative score. The minus sign in $e^{-z}$ becomes plus, so we need $e^{1.4} \approx 4.055$ . Then $p = \frac{1}{1 + 4.055} \approx 0.198$ . Multiplying by 100 gives 19.8 percent.
Compare again, and find the boundary. Since $0.198 < 0.5$ , the model predicts fail for the 2-hour student. The boundary is where the raw score is exactly zero, so $0.8x - 3 = 0$ , which gives $x = \frac{3}{0.8} = 3.75$ hours.

Answer: A student who studies 5 hours has about a 73.1% chance of passing, so the model predicts pass. A student who studies 2 hours has about 19.8%, so the model predicts fail. The 0.5 boundary falls at 3.75 hours of study, where the raw score is zero.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Treating the output as a straight-line prediction. The sigmoid bends the line, so one extra hour does not change the probability by the same amount everywhere.
Expecting the output to be a yes-or-no answer. The result is a probability. You must apply a threshold to turn it into a label.
Dropping the minus sign in the exponent. The formula uses $e^{-z}$ , so a large positive $z$ gives a probability close to 1, and forgetting the sign reverses the answer.
Assuming the word regression means a numeric prediction. Logistic regression is a classifier: it predicts a class, through a probability.

Where you will meet it

Spam filters, which estimate the chance that an email is spam from the words it contains.
Medical risk scores, which estimate the probability of a condition from measurements such as age and blood pressure.
Credit scoring, which estimates the chance that a loan applicant will default.
Baseline models for any yes-or-no prediction, such as whether a customer will click an advertisement.

Questions people ask

How is logistic regression different from linear regression?