Huber Loss

In one sentence: Huber loss scores small misses by squaring them, as squared error does, but scores large misses with a straight-line penalty, so one wildly wrong answer cannot dominate the score.

$L_{\delta}(a) = \begin{cases} \frac{1}{2} a^{2} & \text{if } |a| \le \delta \\ \delta \left(|a| - \frac{1}{2}\delta\right) & \text{otherwise} \end{cases}$

Read it out loud: “L sub delta of a equals one half times a squared, if the size of a is less than or equal to delta. Otherwise, it equals delta times, in brackets, the size of a minus one half of delta. Here a is the miss, the true value minus the prediction. Delta is the Greek letter delta, and it is a threshold, a switch point that you choose.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Piecewise rules and the curly brace: A large left curly brace with two rows is a piecewise formula: one rule with different parts for different situations. Each row has a formula on the left and a condition on the right, such as ‘if $|a| \le \delta$ ’. For each input, check the condition first and then use the one row that matches. Only one row applies to any given input.
The signs $\le$ , $\ge$ and $<$: The sign $\le$ means less than or equal to: the value on the left is smaller than the value on the right, or the two are exactly equal. The sign $\ge$ means greater than or equal to. The plain sign $<$ means strictly smaller, with no equality allowed. So $0.5 \le 1$ is true, and $4 \le 1$ is false.
The Greek letter delta, $\delta$: The lowercase Greek letter delta, said DEL-tuh, is a common name for a small amount or a threshold. In this formula $\delta$ is the switch point: a positive number you choose, in the same units as the data. Misses no bigger than $\delta$ get one treatment and bigger misses get the other. It is a chosen setting, not a measured quantity.
Absolute value $|a|$ and the miss $a$: Two straight bars, $|a|$ , mean the absolute value: the size of $a$ with its sign removed. So $|-4| = 4$ and $|0.5| = 0.5$ . The letter $a$ is the miss, also called the residual: the true value minus the prediction. A miss can be positive, negative or zero, and the absolute value tells you how big it is.
Squares and one half, $a^{2}$ and $\frac{1}{2}$: $a^{2}$ means $a$ times $a$ , the square of $a$ . A square is never negative, because $(-3)^{2} = 9$ the same as $3^{2} = 9$ . The fraction $\frac{1}{2}$ means one half, or 0.5, so $\frac{1}{2} a^{2}$ means: square the miss and then take half of the result. The one half is there so the two pieces of the formula fit together neatly.
Loss, the score a model pays: A loss is a number that measures how bad a model’s predictions are, and lower is better. Training searches for settings that make the total loss as small as possible. Huber loss is one choice of loss, and it gives each miss its own penalty. The average of those penalties over all cases is the score for the whole model.
Outliers: An outlier is a data point far away from the others, such as a travel time that is huge because of an accident, or a typing mistake in a price. Outliers pull scores strongly when the penalty grows with the square of the miss. That is the reason the choice of loss matters when the data contain a few extreme values.

What each symbol means

Symbol	Meaning
$L_{\delta}(a)$	The Huber loss for one miss $a$ : the penalty the model pays for that miss. The subscript $\delta$ shows which threshold was used. It is measured in squared units of the data, such as square minutes for travel times, the same kind of unit that squared error uses.
$a$	The residual, or miss, for one case: the true value minus the prediction, $a = y - \hat{y}$ . It has the unit of the data, such as minutes for a travel-time forecast. It can be positive, negative or zero.
$\delta$	The threshold, the Greek letter delta, a positive number you choose. Misses whose size is no bigger than $\delta$ get the squared treatment, and bigger misses get the straight-line treatment. It has the same unit as $a$ , such as minutes.
$\|a\|$	The absolute value of the miss: its size with the sign removed. For example $\|-4\| = 4$ . It has the same unit as $a$ , so a miss of 4 minutes has size 4 minutes.
$\frac{1}{2}$	One half, the fixed number 0.5. It has no unit. It appears in both pieces, and it is chosen so that the squared piece and the straight-line piece meet at the same height when $\|a\| = \delta$ .
$\text{if}, \text{otherwise}$	The words in the rows of the curly brace. ‘If’ introduces the condition for the squared row, and ‘otherwise’ means the straight-line row applies to every other case. They are words, not numbers, and they tell you which row to use.

The formula piece by piece

$a$: The smallest piece: the miss for one example, the true value minus the prediction. It is the raw input to the Huber rule. Its sign shows whether the prediction was too low or too high, but the rule mostly looks at its size.
$|a|$: The size of the miss, with the sign removed. The rule compares this size with the threshold to pick a row. A miss of 4 minutes and a miss of minus 4 minutes give the same size, so they get the same row.
$\frac{1}{2} a^{2}$: Half the square of the miss, used when the miss is small. For a miss of 0.5 it equals $\frac{1}{2}(0.5)^{2} = 0.125$ . This has the shape of squared error, which grows quickly as the miss grows, but it is used only for small misses.
$\delta\left(|a| - \frac{1}{2}\delta\right)$: The straight-line piece, used when the miss is bigger than the threshold. It grows in step with the size of the miss, not with its square, so a large miss gets a moderate penalty. Subtracting half of $\delta$ makes it join smoothly with the squared piece.
$L_{\delta}(a) = \begin{cases} \frac{1}{2} a^{2} & \text{if } |a| \le \delta \\ \delta \left(|a| - \frac{1}{2}\delta\right) & \text{otherwise} \end{cases}$: The whole rule in one sentence: when the size of a miss is no bigger than the threshold, the penalty is half the square of the miss, and when it is bigger, the penalty is the threshold times the size of the miss minus half the threshold.

What the formula is doing, and why

What goes in and what comes out. The Huber loss takes one miss $a$ and a threshold $\delta$ , and returns one penalty. You compute it for every case, then average the penalties to get the score for the whole model. The threshold is a number you choose in the same unit as the data, such as minutes.

Why two pieces? Mean squared error is smooth and easy to train with, but one very wrong point can dominate it, because squaring makes big misses huge. Mean absolute error treats every size in a straight line, so outliers cannot dominate, but its sharp corner at zero is awkward for some training methods. Huber loss uses the squared piece for small misses and the straight-line piece for large ones. Small misses get the smooth squared treatment, and large misses get a penalty that grows only steadily.

Why do the pieces join smoothly? At the switch point $|a| = \delta$ , the squared piece gives $\frac{1}{2}\delta^{2}$ , and the straight-line piece gives $\delta(\delta - \frac{1}{2}\delta) = \frac{1}{2}\delta^{2}$ . Both pieces reach the same height. Their slopes match there too: the slope of the squared piece is $\delta$ , and the slope of the straight-line piece is also $\delta$ . So the curve has no sharp corner, and training methods that follow slopes work smoothly everywhere.

Choosing delta. A small $\delta$ makes most misses count in a straight line, which resists outliers well but treats small errors less precisely. A large $\delta$ makes almost every miss count as squared, so the loss gets close to mean squared error. The right value depends on what a typical miss looks like in your data. People often try a few values and keep the one that works best on held-out data, meaning test cases the model did not learn from.

An everyday picture

Think of a teacher grading essays. Small slips cost a little, and the cost climbs faster as slips add up. For a truly bad error, the penalty switches to a steady number of points for each unit of badness. One disaster then cannot wipe out the whole grade, while small mistakes still matter.

Worked example: Two travel-time misses with a threshold of 1 minute

A travel-time app predicts two trips. For trip 1 the miss is $a = 0.5$ minutes, and for trip 2 it is $a = 4$ minutes. Use the threshold $\delta = 1$ minute. We find the Huber loss for each trip, then the average. The loss is measured in squared minutes, the same unit that squared error uses.

Decide which piece applies to trip 1. The miss is 0.5 minutes, so its size is $|0.5| = 0.5$ . That is less than or equal to the threshold of 1 minute, so we use the squared piece, $\frac{1}{2} a^{2}$ . A small miss gets the smooth treatment.
Compute the squared piece. We square the miss, $0.5^{2} = 0.25$ , and take half of it: $\frac{1}{2} \times 0.25 = 0.125$ . So trip 1 has a Huber loss of 0.125 squared minutes, a small penalty for a small miss.
Decide which piece applies to trip 2. The miss is 4 minutes, which is bigger than the threshold of 1 minute. So we switch to the straight-line piece, $\delta(|a| - \frac{1}{2}\delta)$ . The size of the miss is 4 and the threshold is 1, so the test $4 > 1$ is true and this row applies.
Compute the straight-line piece. Half of $\delta = 1$ is $0.5$ , and $|a| - 0.5 = 4 - 0.5 = 3.5$ . Multiplying by $\delta = 1$ gives $1 \times 3.5 = 3.5$ . So trip 2 has a Huber loss of 3.5 squared minutes, which is much smaller than the squared penalty would be.
Average the two losses. Adding the two penalties gives $0.125 + 3.5 = 3.625$ . Dividing by 2, the number of trips, gives an average of $1.8125$ squared minutes. Averaging is the last step because the score is meant to describe the whole set of trips.
Compare with squared error. Pure squared error, using the same one half factor, would charge $\frac{1}{2}(4)^{2} = 8$ for trip 2. That is more than twice the Huber penalty of 3.5. Under squared error the one large miss would take over a much bigger share of the score.

Answer: The average Huber loss is 1.8125 squared minutes. The trip with the 4-minute miss costs 3.5 instead of 8, so that one large miss does not take over the score, while the small miss is still scored smoothly as a squared penalty.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Using the squared piece for every miss. Huber loss switches to the straight-line piece once the size of the miss is bigger than $\delta$ .
Dropping the one half in the squared piece. Without it the two pieces no longer meet at the switch point, so the loss would jump there.
Choosing $\delta$ without looking at the data. The right threshold depends on the size of typical misses, and the same number means very different things in minutes and in dollars.
Mixing up the units of $\delta$ and $a$ . The threshold must be in the same unit as the misses, or the switch point lands in the wrong place.

Where you will meet it

Robust regression, a way of fitting a line to data, where a few outliers should not pull the line too far.
Training object detectors, which are models that find objects in photos and predict the corners of a box around each one. A close relative of Huber loss, called smooth L1 loss, is common there.
Reinforcement learning, a way of training a program from rewards by trial and error, where Huber loss is often used to keep rare, large surprises from dominating the update.
Predicting prices or durations in data that contain a few extreme values.

Questions people ask

What is a good value for delta?

There is no single right value. Pick it in the same units as your data, based on what counts as a normal miss. Many people try several values and keep the one that works best on held-out data, meaning test cases the model did not learn from.

Is Huber loss the same as smooth L1 loss?

They are closely related. Smooth L1 uses the same two pieces with $\delta = 1$ , so it is a common special case of the Huber loss.

Why not use absolute error instead?

Absolute error has a sharp corner at zero, which makes training that follows slopes less smooth. Huber loss keeps the resistance to large misses and stays smooth near zero.

Mean Absolute Error (MAE): Mean absolute error is the average size of a model’s misses, ignoring whether each miss was too high or too low, so one very large miss cannot dominate the score.
Mean Squared Error (MSE): Mean squared error measures how wrong a model’s guesses are by squaring each miss, so big misses count much more, and then averaging all of the squared misses.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.

Sources

Published 2026-10-22. Plain-text copy for AI agents and readers: ai-ml/huber-loss.md.