L2 Regularization

In one sentence: L2 regularization adds a penalty for large weights, so the model prefers small, moderate weights that tend to hold up on new data.

$J(w) = L(w) + \lambda \sum_{j} w_j^{2}$

Read it out loud: “J of w equals L of w, plus lambda times the sum, over j, of w sub j squared. Here J of w is the total cost that training tries to make small, and L of w is the data loss, which measures how wrong the model is on the training examples. Lambda is the Greek letter lambda, a strength you choose, and w sub j is weight number j, squared.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Weights, the bias, and linear regression: A weight is a number that scales one input before the model adds the results. A linear regression model predicts a number as a weighted sum of its inputs, plus a bias, an extra number added at the end that sets the level of the output. A model with two inputs has two weights, $w_1$ and $w_2$ . A big weight means its input has a strong effect on the prediction, and a weight near zero means the input barely matters.
Subscripts and the sum sign, $w_j$ and $\sum_{j}$: The small $j$ in $w_j$ is a subscript that counts the weights: $w_1$ is the first weight and $w_2$ is the second. The sign $\sum_{j}$ is the capital Greek letter sigma, which means add up the expression for every $j$ . So $\sum_{j} w_j^{2}$ means: square each weight, then add all the squares together.
Squares, $w_j^{2}$: $w_j^{2}$ is the square of $w_j$ , the number times itself. Squaring makes large numbers grow fast: $1^{2} = 1$ , $2^{2} = 4$ and $4^{2} = 16$ . It also removes the minus sign, because $(-0.5)^{2} = 0.25$ . So the square of a weight measures its size without caring whether it is positive or negative.
The Greek letter lambda, $\lambda$: The Greek letter lambda, said LAM-duh, is a common name for a strength or a rate. Here $\lambda$ is a number you choose, zero or more. It sets how strongly the penalty counts against the data loss. Zero means no penalty at all, and a bigger value means stronger pressure toward small weights.
Loss, data loss and total cost: A loss is a number that measures how wrong a model is, and lower is better. The data loss, written $L(w)$ , is the loss on the training examples when the weights are $w$ . The total cost, written $J(w)$ , adds a penalty to the data loss. Training searches for weights that make the total cost as small as possible.
Overfitting: Overfitting means a model learns the random noise in its training examples as if it were a real pattern. It scores well on the examples it has already seen and badly on new examples. Regularization, which adds a penalty, is one way to reduce overfitting.

What each symbol means

Symbol	Meaning
$J(w)$	The total cost that training tries to make small: the data loss plus the penalty. It has no unit in this toy example, and smaller is better. Writing $(w)$ tells you it depends on the whole list of weights.
$L(w)$	The data loss, a measure of how wrong the model is on the training examples when its weights are $w$ . It is a pure number with no unit here. Lower values mean the predictions fit the training examples better. It can be any loss, such as squared error or cross-entropy.
$\lambda$	The regularization strength, the Greek letter lambda, a number you choose that is zero or more. It has no unit. Zero means no penalty, and bigger values pull the weights harder toward zero.
$\sum_{j}$	The sum sign: add up the expression to its right for every weight, starting from weight 1 and continuing to the last weight. It has no unit and tells you to total the squared weights in one step.
$w_j^{2}$	The square of weight number $j$ . A weight is a pure number with no unit in this toy example, and its square is never negative. A weight of 2 has the square 4, and a weight of minus 0.5 has the square 0.25.
$w_j$	One weight, the number that scales input number $j$ . Training sets its value. It has no unit in this toy example. In real models the unit of a weight depends on the unit of the input it scales.
$w$	The whole list of weights, $w = (w_1, w_2, \ldots)$ . The three dots mean the list continues for as many weights as the model has. Training adjusts all of them together, so think of $w$ as one bundle of numbers.

The formula piece by piece

$w_j$: One weight, the number that scales input $j$ . In a model with two inputs there are two weights, $w_1$ and $w_2$ . A large weight gives its input a strong say in the prediction.
$w_j^{2}$: The square of one weight. Squaring makes large weights much more costly than small ones. A weight of 4 has the square 16, while a weight of 1 has the square 1, so the weight of 4 costs 16 times as much.
$\lambda \sum_{j} w_j^{2}$: The penalty. First square every weight and add the squares, which gives the overall size of the weights. Then multiply by $\lambda$ , which sets how heavily the penalty counts in the total cost.
$L(w) + \lambda \sum_{j} w_j^{2}$: The total cost: the data loss, which rewards a good fit, plus the penalty, which charges for big weights. Training has to balance these two parts against each other.
$J(w) = L(w) + \lambda \sum_{j} w_j^{2}$: The whole formula in one sentence: the total cost equals the data loss plus lambda times the sum of the squared weights, so training pays both for a poor fit and for large weights.

What the formula is doing, and why

The problem it solves. A model fitted only to its training examples can memorize noise. With enough weights it can draw a wiggly curve through every training point, using very large weights that chase each bump. Such a model scores well on the examples it has seen and badly on new examples. That gap is called overfitting, and regularization is the usual cure.

What it adds. The L2 penalty, $\lambda \sum_{j} w_j^{2}$ , is added to the data loss. Read it in three moves: square each weight, add up the squares, and multiply the total by $\lambda$ . Large weights now cost a lot in the total cost, so training has to balance two goals: fitting the data, and keeping the weights small.

Why square? Squaring makes the penalty grow fast for big weights. A weight of 4 costs 16 times as much as a weight of 1, so the pressure falls hardest on the largest weights. Small weights cost almost nothing, so the penalty mostly targets the big ones. During training this shows up as weight decay: each update shrinks every weight a little toward zero before the data pulls it back where it needs to be.

The main choice, and when it applies. The strength $\lambda$ is the key decision. If $\lambda = 0$ , you get the plain model with no penalty. If $\lambda$ is too big, the weights are squeezed so hard that the model underfits and misses real patterns, even on the training data. Try several values and keep the one that does best on validation data, which is a set of examples kept aside to check the choice. For linear regression, the L2 version is called ridge regression.

An everyday picture

Think of a student who studies by memorizing last year’s answer key. A tutor adds a rule: any study method that relies on a few very strong shortcuts costs points. The student now spreads effort more evenly across the material, so the skill carries over to new questions. The penalty stops the model from leaning too hard on any one input.

Worked example: Choosing between two delivery-time models

Two models predict how many minutes a delivery takes, from two inputs: the distance and the number of stops. This is a hypothetical example with made-up weights. Each model has one weight per input. Model A has weights $w = (2, -0.5)$ and data loss $L = 0.30$ . Model B has weights $w = (1, -0.5)$ and data loss $L = 0.35$ . The penalty strength is $\lambda = 0.1$ . Which model has the lower total cost $J(w)$ ?

Square the weights of Model A. The penalty is built from squares, so we square each weight first: $2^{2} = 4$ and $(-0.5)^{2} = 0.25$ . Squaring removes the minus sign, so both weights count by their size only. Their squares add up to $4 + 0.25 = 4.25$ .
Multiply the squared total by lambda. The penalty is $\lambda$ times the sum of the squares: $0.1 \times 4.25 = 0.425$ . Most of this penalty comes from Model A’s large weight of 2, because squaring makes a weight of 2 count four times as much as a weight of 1.
Add the data loss for Model A. The total cost adds the data loss, $0.30$ , to the penalty: $J = 0.30 + 0.425 = 0.725$ . Model A has the lowest data loss of the two models, but the penalty pushes its total cost up.
Square the weights of Model B. Model B has weights $1$ and $-0.5$ . Their squares are $1^{2} = 1$ and $(-0.5)^{2} = 0.25$ , which add up to $1.25$ . Model B’s largest weight is only 1, so its squared total is far smaller than Model A’s total of 4.25.
Compute Model B’s penalty and total cost. The penalty is $0.1 \times 1.25 = 0.125$ . Adding the data loss of $0.35$ gives $J = 0.35 + 0.125 = 0.475$ . Model B fits the training data a little less well, but its smaller weights keep its total cost lower.
Compare the two totals. Model B has the total cost 0.475, and Model A has 0.725. Under this rule the lower total is preferred, so Model B wins, even though its data loss is higher. The penalty made the difference.

Answer: Model B has the lower total cost, 0.475 against 0.725. Model A fits the training data slightly better, but its large weight costs more in penalty than its better fit saves, so the L2 rule prefers the smaller weights of Model B.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Setting $\lambda$ too large. A huge penalty crushes all the weights toward zero, and the model underfits, doing badly even on the training data.
Penalizing the bias term along with the weights. The bias sets the level of the output and is usually left out of the penalty.
Comparing total costs as if they were pure data losses. The total includes the penalty, so compare the data losses separately when you want to judge how well a model fits.
Skipping input scaling. The penalty treats every weight equally, so inputs measured on very different scales are penalized unevenly. Distance in meters gives a much smaller weight than distance in kilometers, so the same penalty treats them very differently.

Where you will meet it

Ridge regression, the version of linear regression with an L2 penalty, which stays stable when inputs are strongly related to each other.
Neural networks, where the same penalty is added during training under the name weight decay.
Logistic regression and linear support vector machines, two classic classifiers that sort examples into groups, where the penalty keeps the weights moderate.
Smoothing flexible curve fits, where the penalty stops the curve from wiggling through every point.

Questions people ask

What is the difference between L2 and L1 regularization?

L2 penalizes squared weights and shrinks all of them toward zero, usually without making any exactly zero. L1 penalizes the sizes of the weights and can push some of them to exactly zero.

Why is it called weight decay?

Each training step multiplies the weights by a factor slightly less than 1 before the data update. Over time the weights decay toward zero unless the data keeps them large.

Is a larger lambda always better?

No. A lambda that is too large underfits, and one that is too small allows overfitting. The best value sits between those extremes and is found with validation data.

L1 Regularization: L1 regularization adds a penalty that grows with the size of each weight, which can drive the weights of unneeded inputs to exactly zero and so remove those inputs.
Linear Regression: Linear regression predicts a number by drawing the straight line that sits closest to the data points, so you can read off a guess for any new input.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.