Hinge Loss

Q: Why are the labels $-1$ and $+1$ instead of 0 and 1?

The product $y f(x)$ needs the sign of the label to match the sign of the score. With labels of $-1$ and $+1$, a correct score gives a positive product and a wrong score gives a negative product.

Q: How is hinge loss different from logistic loss?

Hinge loss is exactly zero for any example beyond the margin, so it ignores easy examples. Logistic loss never reaches zero, so it keeps pushing a little on every example. Logistic loss is also the choice that produces probabilities.

Q: Why is the margin exactly 1?

The number 1 is a choice of scale. If you multiply the weights and the bias by the same positive number, every score is multiplied too, so the margin can always be set to 1 by rescaling. What matters is the width of the gap between the boundary and the nearest examples.

In one sentence: Hinge loss charges nothing for an answer that is correct and clearly on its own side of the line, and a growing penalty for answers that are wrong or too close to the line.

$L = \max(0,\ 1 - y\, f(x))$

Read it out loud: “L equals the larger of zero and one minus y times f of x. Here L is the loss for one example, y is the true label, which is plus 1 or minus 1, and f of x is the model's raw score for the input x. The word max means take the larger of the two numbers in the brackets.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Labels $+1$ and $-1$: A label is the true answer for an example. This formula names the two classes $+1$ and $-1$ . For example, spam could be $+1$ and not spam could be $-1$ . These two numbers are chosen because multiplying a label by a score gives a positive result when the score points the right way, and a negative result when it points the wrong way.
The raw score $f(x)$: The model computes a raw score, written $f(x)$ , for an input $x$ . A score is only a number, not a probability. Its sign shows the side of the line: positive points toward $+1$ and negative points toward $-1$ . Its size shows how far the input sits from the line, so a score of 2.0 is more confident than a score of 0.3.
Features, the input $x$: The input $x$ is a list of features, which are numbers that describe one example. For an email, the features could be how many times each word appears. The model combines the features into one score. The features are the raw description of the example, and the score is the model’s single answer based on them.
Classifiers and the decision boundary: A classifier puts each input into one of two or more classes. A decision boundary is the dividing line, or surface, between the areas for the classes. Inputs on one side get the label $+1$ , and inputs on the other side get $-1$ . A support vector machine is a classifier that picks the boundary with the widest possible gap to the nearest examples.
The margin, the number 1, and the product $y f(x)$: Writing $y f(x)$ means multiply the label $y$ by the score $f(x)$ . The result is positive when the score has the same sign as the label, which means a correct guess. The margin is the target value of 1: a correct example should reach a product of at least 1, which means it is on its own side with room to spare.
The maximum, $\max(0, \cdot)$: $\max(a, b)$ returns the larger of two numbers. So $\max(0, 0.7) = 0.7$ , and $\max(0, -1.0) = 0$ . Any negative value is replaced by 0. This keeps the penalty from going below zero, so an example with plenty of room to spare contributes nothing instead of earning a negative penalty.
Loss and convexity, a bowl with one bottom: A loss is a number that scores how bad a prediction is, and training searches for settings that make the total loss small. A function is convex when its graph is shaped like a bowl, with one lowest point and no separate dips where training could get stuck. Following the downhill direction on a convex loss tends to reach the lowest point.

What each symbol means

Symbol	Meaning
$L$	The hinge loss for one example, the penalty that example adds to the total. It is a pure number with no unit, and it is never below zero. It is exactly zero for an example that is correct and beyond the margin, and it grows as the example gets more wrong.
$y$	The true label of the example, either $+1$ or $-1$ . It is a pure number with no unit. It names which of the two classes is correct, for example spam as $+1$ and not spam as $-1$ .
$f(x)$	The raw score the model gives the input $x$ . It is a pure number with no unit. Its sign picks a class, and its size shows how confident the model is. A score of 2.0 is more confident than a score of 0.3, but neither one is a probability.
$x$	The input example, the list of features that describes one item, such as the word counts in one email. It is the raw information the model starts from, and the model turns it into a single score.
$1$	The margin: a correct example should reach a product of at least 1 with its own label. It is a fixed number with no unit, and it sets the scale of the scores. Scores short of 1 still count as correct but get a small penalty.
$\max$	The maximum sign: it picks the larger of the numbers inside the brackets. Here the brackets hold 0 and $1 - y f(x)$ , so any negative shortfall becomes 0. The maximum has no unit and keeps the penalty from going below zero.

The formula piece by piece

$y f(x)$: The label times the score. It is positive when the score has the same sign as the label, which means a correct guess. For example, a label of $-1$ and a score of $0.5$ give $-1 \times 0.5 = -0.5$ , which is negative, a sign of a wrong side.
$1 - y f(x)$: How far the example falls short of the margin of 1. If $y f(x) = 2$ , the answer is $1 - 2 = -1$ , which means the example is past the margin with room to spare. If $y f(x) = 0.3$ , the answer is $0.7$ , which means the example is 0.7 short of the margin.
$\max(0,\ 1 - y\, f(x))$: The shortfall, but never below zero. Examples with room to spare contribute 0, and examples short of the margin contribute their shortfall. A wrong-side example has a shortfall bigger than 1 and so gets the largest penalty.
$L = \max(0,\ 1 - y\, f(x))$: The whole formula in one sentence: the penalty for one example is the amount it falls short of the margin of 1, or zero when it has already cleared the margin.

What the formula is doing, and why

What goes in and what comes out. A support vector machine learns a dividing line between two classes. For each training example we know the true label $y$ , which is $+1$ or $-1$ , and the model computes a raw score $f(x)$ . Hinge loss takes $y$ and $f(x)$ and returns one penalty. Summing the penalties over all training examples gives the total that training tries to make small.

Why multiply the label by the score? A correct guess means the score has the same sign as the label, so $y f(x)$ is positive. A wrong guess flips the sign, so $y f(x)$ is negative. Multiplying turns the question ‘are the signs the same?’ into one number that the formula can use directly. A bigger positive product means a more confident correct guess.

Why the margin of 1? The formula asks for more than a correct sign. It wants $y f(x) \ge 1$ , which means the score reaches at least 1 in size on its own side. An example that is right but inside that margin, with $0 < y f(x) < 1$ , still gets a small penalty. An example on the wrong side, with $y f(x) < 0$ , gets a penalty bigger than 1 that grows as it goes further wrong. The margin keeps the dividing line a clear distance from every example.

Why it works, and its limits. Examples beyond the margin contribute exactly zero, so training is shaped by the hard cases near the line. The loss is convex, a bowl with one bottom, so training does not get trapped in false low spots. It has a sharp corner at $y f(x) = 1$ , where the slope jumps, which makes it less smooth than squared error. It gives scores, not probabilities. Use it when you want a clear boundary, and use logistic loss when you want probabilities.

An everyday picture

Think of a line judge in tennis who decides whether each ball is in or out. A ball that lands well inside the line costs nothing. A ball that lands close to the line is a small concern. A ball that lands outside is a fault, and the farther outside it lands, the bigger the fault. Hinge loss judges each example in the same way, asking for a clear margin rather than a narrow pass.

Worked example: Three spam-filter scores near the boundary

A spam filter gives each of three emails a raw score. Spam has the label $+1$ and not spam has the label $-1$ . Email 1 is spam with score $f(x) = 2.0$ . Email 2 is spam with score $0.3$ . Email 3 is not spam with score $0.5$ . The scores are unitless. We find the hinge loss for each email, then the average.

Email 1: compute the product. The label is $+1$ and the score is $2.0$ , so $y f(x) = (+1)(2.0) = 2.0$ . The product is positive, which means the sign matches and the filter is right. Its size is bigger than 1, so this email already has room to spare beyond the margin.
Email 1: compute the penalty. The shortfall is $1 - 2.0 = -1.0$ , a negative number because the email is past the margin. The rule takes the larger of 0 and $-1.0$ , which is 0. So email 1 costs nothing, and the training process has no reason to move the boundary because of it.
Email 2: compute the product. The label is $+1$ and the score is $0.3$ , so $y f(x) = 0.3$ . The sign matches, so the guess is correct, but the value is below 1. That means the email sits inside the margin, too close to the dividing line for comfort.
Email 2: compute the penalty. The shortfall is $1 - 0.3 = 0.7$ . The larger of 0 and 0.7 is 0.7, so email 2 costs 0.7. It is on the correct side, but the penalty pushes training to give it more room from the line.
Email 3: compute the product. The label is $-1$ , meaning not spam, and the score is $0.5$ , which leans toward spam. So $y f(x) = (-1)(0.5) = -0.5$ . The product is negative, so the sign is wrong: the filter would put a real message on the spam side.
Email 3: compute the penalty. The shortfall is $1 - (-0.5) = 1.5$ . This is positive, so $\max(0, 1.5) = 1.5$ . Email 3 costs 1.5, the largest of the three, because it is on the wrong side and also short of the margin.
Average the three penalties. Adding the penalties gives $0 + 0.7 + 1.5 = 2.2$ . Dividing by the three emails gives $\frac{2.2}{3} \approx 0.733$ . This is the average hinge loss for these three examples.

Answer: The average hinge loss is about 0.733. Email 1 costs nothing, email 2 is correct but too close to the line and costs 0.7, and email 3 is on the wrong side and costs 1.5. Training works to reduce the largest penalties first, so email 3 matters most here.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Using labels of 0 and 1. Hinge loss needs labels of $-1$ and $+1$ . With 0 and 1, multiplying the label by the score no longer tests the sign, and the results are wrong.
Leaving out the $\max(0, \cdot)$ step. Without it, an example with a large margin would get a negative penalty, which rewards it without any limit.
Reading the raw score as a probability. The score $f(x)$ is not between 0 and 1, and hinge loss does not produce probabilities.
Assuming a loss of 0 means the model is perfect. Zero means only that this example is correct and beyond the margin, not that every example is.

Where you will meet it

Training linear support vector machines, a classic method for sorting data into two groups, where hinge loss is the standard loss.
Kernel support vector machines, which apply the same loss after transforming the inputs into a richer set of features.
Ranking and retrieval models, which learn to score a matching pair above a non-matching pair by a set margin.
Some neural network classifiers, which use hinge loss at the output in place of cross-entropy loss.

Questions people ask

Why are the labels $-1$ and $+1$ instead of 0 and 1?

The product $y f(x)$ needs the sign of the label to match the sign of the score. With labels of $-1$ and $+1$ , a correct score gives a positive product and a wrong score gives a negative product.

How is hinge loss different from logistic loss?