L1 Regularization

In one sentence: L1 regularization adds a penalty that grows with the size of each weight, which can drive the weights of unneeded inputs to exactly zero and so remove those inputs.

$J(w) = L(w) + \lambda \sum_{j} |w_j|$

Read it out loud: “J of w equals L of w, plus lambda times the sum, over j, of the absolute value of w sub j. Here J of w is the total cost that training tries to make small, and L of w is the data loss, which measures how wrong the model is on the training examples. Lambda is the Greek letter lambda, a strength you choose. The absolute value of w sub j is the size of weight number j, with its sign removed.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Weights, the bias, and linear regression: A weight is a number that scales one input before the model adds the results. A linear regression model predicts a number as a weighted sum of its inputs, plus a bias, an extra number added at the end that sets the level of the output. A model with many inputs has many weights, one for each input. A weight of exactly zero means the model ignores that input completely.
Subscripts and the sum sign, $w_j$ and $\sum_{j}$: The small $j$ in $w_j$ is a subscript that counts the weights: $w_1$ is the first weight and $w_2$ is the second. The sign $\sum_{j}$ is the capital Greek letter sigma, which means add up the expression for every $j$ . So $\sum_{j} |w_j|$ means: take the size of each weight, then add all the sizes together.
Absolute value bars, $|x|$: Two straight bars around a number, $|x|$ , mean its absolute value: its size with the sign removed. So $|-0.5| = 0.5$ , $|2| = 2$ and $|0| = 0$ . Unlike a square, the absolute value grows in a straight line: $|4| = 4$ is exactly twice $|2| = 2$ . That straight-line growth is the key difference from the L2 penalty.
The Greek letter lambda, $\lambda$: The Greek letter lambda, said LAM-duh, is a common name for a strength or a rate. Here $\lambda$ is a number you choose, zero or more. It sets how strongly the penalty counts against the data loss. Zero means no penalty at all, and a bigger value means stronger pressure toward zero.
Feature selection and overfitting: A feature is one input column in the data, such as the distance of a delivery or its number of stops. Feature selection means choosing which inputs to keep. Overfitting means a model learns the random noise in its training examples as if it were a real pattern, and it then does badly on new examples. A model that keeps only the useful inputs is simpler and tends to overfit less.
A corner in a graph: A corner is a point where a curve turns sharply, like the tip of the letter V. At a corner the slope has no single value: it is one slope on the left and a different one on the right. The absolute value has a corner at zero, which is why training needs extra care there.

What each symbol means

Symbol	Meaning
$J(w)$	The total cost that training tries to make small: the data loss plus the penalty. It has no unit in this toy example, and smaller is better. Writing $(w)$ tells you that it depends on the whole list of weights.
$L(w)$	The data loss, a measure of how wrong the model is on the training examples when its weights are $w$ . It is a pure number with no unit here. Lower values mean the predictions fit the training examples better.
$\lambda$	The regularization strength, the Greek letter lambda, a number you choose that is zero or more. It has no unit. Zero means no penalty, and bigger values pull the weights harder toward zero, removing more of them.
$\sum_{j}$	The sum sign: add up the expression to its right for every weight, from the first weight to the last. It has no unit and tells you to total the sizes of all the weights in one step.
$\|w_j\|$	The absolute value of weight number $j$ : its size with the sign removed. For example $\|-0.5\| = 0.5$ and $\|2\| = 2$ . A weight of 2 costs twice as much as a weight of 1, and a negative weight costs the same as a positive weight of the same size.
$w$	The whole list of weights, $w = (w_1, w_2, \ldots)$ . The three dots mean the list continues for as many weights as the model has. Training adjusts all of them together, so think of $w$ as one bundle of numbers.

The formula piece by piece

$w_j$: One weight, the number that scales input $j$ . Training adjusts it. A weight of exactly zero means the model ignores input $j$ completely, and that input is then effectively removed.
$|w_j|$: The size of one weight, with its sign removed. A weight of minus 0.5 has size 0.5, and it gets the same penalty as a weight of plus 0.5. This is the only change from the L2 idea, which squares the weight instead.
$\sum_{j} |w_j|$: The total size of all the weights, found by taking the absolute value of each weight and adding everything together. This total is called the L1 norm of the weights.
$\lambda \sum_{j} |w_j|$: The penalty: the total size of the weights multiplied by $\lambda$ , which sets how strongly the penalty counts. Each unit of weight costs the same flat amount, whether the weight is big or small.
$J(w) = L(w) + \lambda \sum_{j} |w_j|$: The whole formula in one sentence: the total cost equals the data loss plus lambda times the sum of the absolute sizes of the weights, so training pays both for a poor fit and for every unit of weight.

What the formula is doing, and why

What goes in and what comes out. Like L2, L1 regularization adds a penalty to the data loss, so training balances fitting the data against keeping the weights small. The total cost is $J(w) = L(w) + \lambda \sum_{j} |w_j|$ . The data loss $L(w)$ measures the fit, and $\lambda$ sets the strength of the penalty. The sum of the absolute weights is called the L1 norm of the weight list.

How it differs from L2. L2 charges by the square, so the charge for a small weight is tiny and the pull toward zero fades as the weight shrinks. L1 charges by the absolute value, so every nonzero weight feels the same steady pull toward zero, whether it is large or small. The squared penalty has a smooth bowl at zero, while the absolute penalty has a sharp V shape at zero.

Why weights can become exactly zero. The steady pull can push a weight all the way to exactly zero. Once a weight reaches zero and the data does not pull it back out, it stays there, and the input it multiplies has no effect. L1 therefore acts as feature selection: the inputs that keep nonzero weights are the ones the data really needs. Under L2, weights get close to zero but rarely land on it.

The cost of the sharp corner, and the choice of lambda. The V-shaped corner at zero means the slope jumps from $-1$ to $+1$ . Training handles this with a subgradient, which is a generalized slope, or with a proximal step, which pushes each weight toward zero by a fixed amount. For linear regression, the L1 version is the lasso, short for least absolute shrinkage and selection operator. The strength $\lambda$ still matters: too large and every weight is zeroed, too small and nothing is removed.

An everyday picture

Think of a sports team where every player on the field costs the same flat fee per minute, however good the player is. The coach keeps the players whose contribution is worth the fee and benches the rest, so some players end up with zero minutes. L1 works the same way on weights: each unit of weight costs the same, so weak inputs can drop to exactly zero.

Worked example: The same two delivery-time models with the L1 penalty

Two models predict how many minutes a delivery takes, from two inputs: the distance and the number of stops. This is a hypothetical example with made-up weights. Model A has weights $w = (2, -0.5)$ and data loss $L = 0.30$ . Model B has weights $w = (1, -0.5)$ and data loss $L = 0.35$ . The penalty strength is $\lambda = 0.1$ . We compare the total costs when the penalty adds up the sizes of the weights.

Find the sizes of Model A’s weights. The size of the weight 2 is $|2| = 2$ , and the size of the weight $-0.5$ is $|-0.5| = 0.5$ . We drop the minus sign because the L1 penalty charges for how big a weight is, not for its direction.
Add the sizes. The total size is $2 + 0.5 = 2.5$ . Unlike the squared total used by L2, this number grows in a straight line as each weight grows, so the weight of 2 costs only twice as much as a weight of 1.
Multiply the total size by lambda. The penalty is $0.1 \times 2.5 = 0.25$ . This is a flat charge on each unit of weight, so Model A pays 0.25 for its weights.
Add the data loss for Model A. The total cost is $J = 0.30 + 0.25 = 0.55$ . Model A’s total cost is 0.55 under the L1 rule.
Find the sizes of Model B’s weights. The size of the weight 1 is $|1| = 1$ , and the size of $-0.5$ is $|-0.5| = 0.5$ . Their total is $1 + 0.5 = 1.5$ , which is less than Model A’s total of 2.5.
Compute Model B’s penalty and total cost. The penalty is $0.1 \times 1.5 = 0.15$ . Adding the data loss gives $J = 0.35 + 0.15 = 0.50$ , so Model B’s total cost is 0.50.
Compare the two totals. Model B has the lower total cost, 0.50 against 0.55. The gap is only 0.05. Under the squared penalty of L2 the gap was 0.25, which is larger, because the square makes the big weight of Model A cost much more.

Answer: Model B has the lower total cost, 0.50 against 0.55. Under the L1 penalty the gap is 0.05, narrower than the 0.25 gap under L2, because each unit of weight costs the same flat amount and does not grow with a square.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Squaring the weights, which turns the penalty into the L2 penalty. The L1 penalty uses absolute values, with no square.
Expecting L1 to shrink every weight by the same ratio. It reduces each nonzero weight by a fixed amount and can remove weak weights entirely.
Putting the bias term in the penalty. As with L2, the bias is usually left out of the penalty.
Reading a zero weight as proof that the input is useless. Correlated inputs can compete, so L1 may keep one of two inputs that carry the same information and set the other to zero.

Where you will meet it

The lasso, the version of linear regression with an L1 penalty, used when you want a model that depends on only a few inputs.
Feature selection on data with thousands of columns, such as gene measurements or word counts, where most inputs are irrelevant.
Pruning neural networks, which means setting many weights to exactly zero so that the network becomes smaller and faster.
Compressed sensing, a method for rebuilding a signal such as an image from fewer measurements than usual, using the L1 penalty to favor simple answers.

Questions people ask

Why does L1 set weights to exactly zero but L2 usually does not?

The L1 penalty has a constant pull away from zero, so the pull stays the same as a weight gets small and can reach zero. The L2 pull fades as the weight shrinks, so weights get close to zero without landing on it.

Should I use L1 or L2?

Use L1 when you suspect many inputs are useless and you want a smaller model. Use L2 when you expect many inputs to matter a little and you want stable, shrunken weights. Testing both on validation data is common practice.

What is the lasso?

The lasso is linear regression with an L1 penalty. Its name stands for least absolute shrinkage and selection operator, which describes both the shrinking of the weights and the selection of inputs.

L2 Regularization: L2 regularization adds a penalty for large weights, so the model prefers small, moderate weights that tend to hold up on new data.
Linear Regression: Linear regression predicts a number by drawing the straight line that sits closest to the data points, so you can read off a guess for any new input.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.