ReLU Activation

In one sentence: ReLU keeps positive numbers unchanged and replaces every negative number with zero, like a switch that is on when the input is positive and off otherwise.

$\text{ReLU}(z) = \max(0, z)$

Read it out loud: “ReLU of z equals the larger of zero and z. ReLU is short for rectified linear unit. The letter z is the input number, which is usually a weighted sum plus a bias, and max means take the bigger of the two numbers.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Positive, negative and zero on the number line: Numbers to the right of zero on the number line are positive, and numbers to the left are negative. Zero sits between them and is neither. A negative number is a value below zero, such as a temperature of minus 5 degrees Celsius. Multiplying two negative numbers gives a positive number, and adding a negative number is the same as subtracting its size.
The maximum $\max(a, b)$: $\max(a, b)$ means ‘the bigger of the two numbers $a$ and $b$ ’. So $\max(0, 4) = 4$ , because 4 is bigger than 0, and $\max(0, -3) = 0$ , because 0 is bigger than $-3$ . The notation picks one of the two numbers and throws the other away, which is exactly what the ReLU rule does to a negative input.
An activation function: An activation function is a rule that a neuron applies to its total before passing the result on. It decides how strong the output is. Without a rule like this, a stack of neuron layers would combine into one straight-line formula. The activation function adds a bend, which is what lets the network represent curved patterns.
Straight lines, slope and linear: A straight line has a constant slope, the amount it rises for each step to the right. The line $y = z$ has slope 1, because each step to the right rises by one step up. A rule is called linear when it is a straight line in this sense. ReLU is not linear overall, because it is a straight line with slope 1 on the positive side and a flat line at 0 on the negative side, which makes a bend.
Gradient, the slope used for learning: The gradient of a function is how much its output changes when its input changes a tiny bit. During training, gradients tell the network which way to adjust its weights. For ReLU, the gradient is 1 on the positive side, because the output follows the input, and 0 on the negative side, because the output stays fixed at zero no matter how the input moves.

What each symbol means

Symbol	Meaning
$\text{ReLU}(z)$	The rectified linear unit output: the value the neuron passes on to the next layer. It is never negative, because negative inputs become 0. It has no unit in most networks, and it is a plain number like the input.
$z$	The input to the activation, usually the weighted sum of a neuron’s inputs plus its bias. It can be any real number, positive, negative or zero. In the example it is a made-up score that says how much the fan should run, and it has no unit.
$\max(0, z)$	The larger of the two numbers 0 and $z$ . If $z$ is positive, the result is $z$ itself, passed on unchanged. If $z$ is negative, the result is 0. This is the entire rule, written as one comparison.

The formula piece by piece

$z = w_1 x_1 + w_2 x_2 + b$: Start with the neuron’s weighted sum. Each input is multiplied by its weight, the products are added, and the bias is added last. This total $z$ is the number ReLU will read, and it can be positive or negative depending on the inputs.
$\max(0, z)$: Compare the total with 0 and keep the bigger one. A positive total is kept exactly as it is. A negative total loses the comparison and becomes 0, so the neuron outputs nothing for that input.
$\text{ReLU}(z) = \max(0, z)$: The whole formula in one sentence: the ReLU output of $z$ equals the larger of zero and $z$ , so positive values pass through unchanged and negative values become zero.

What the formula is doing, and why

An activation function sits at the output of a neuron and decides what gets passed on. A neuron first computes a weighted sum $z$ of its inputs, then applies the activation. Without a non-linear activation, a stack of layers collapses into one big linear formula, and that formula cannot learn curved patterns.

What does ReLU do? ReLU, short for rectified linear unit, is the simplest useful choice, and it has two cases. If $z$ is positive, the output equals $z$ , a straight line with slope 1. If $z$ is zero or negative, the output is 0. The output never goes below zero. The bend at zero is the only non-straight part, and that bend is enough to let stacked layers learn curved patterns.

Why is it popular? The main advantage is cheap computation. Producing the output takes one comparison, with no exponentials. For positive inputs the slope is always 1, so the learning signal does not shrink there the way it can with curves such as the sigmoid, which flatten out at both ends. That is a large part of why deep networks train well with it.

What is the known drawback? The dying ReLU problem. A neuron whose input stays negative outputs 0 and receives a gradient of 0, so its weights stop changing, and it can stop learning for good. Variants such as the leaky ReLU keep a small slope on the negative side to avoid this.

An everyday picture

Think of a one-way valve on a water pipe. Water pushing forward flows through unchanged, so a bigger push gives a bigger flow. Water pushing backward meets a closed valve and stops at zero. The neuron lets through only the part of the signal that points in the forward direction.

Worked example: A smart fan that depends on the window

A smart fan uses one neuron to decide how hard to run. Its inputs are $x_1 = 3$ , the number of degrees the room is above a comfortable temperature, and $x_2$ , which is 1 when a window is open and 0 when it is shut. The weights are $w_1 = 0.5$ per degree and $w_2 = -2$ for the open window, because open windows cool the room. The bias is $b = 0.25$ . The fan’s strength is on a made-up scale with no unit. We find the output with the window open, then with the window shut.

Compute the weighted sum with the window open. Here $x_2 = 1$ . The first input contributes $0.5 \times 3 = 1.5$ , and the open window contributes $(-2) \times 1 = -2$ . Those two products are added, which gives $1.5 - 2 = -0.5$ , so far the signal points toward turning the fan off.
Add the bias. The bias $0.25$ is added to the weighted sum, so $z = -0.5 + 0.25 = -0.25$ . The total is negative, which means the room would be cooled by the open window, so the fan should not work hard.
Apply ReLU to the negative total. ReLU compares the total with zero, $\max(0, -0.25) = 0$ . The negative total loses, so the output is 0. The fan is off, and the neuron has switched itself off for this input.
Now shut the window. With the window closed, $x_2 = 0$ , so the window term is $(-2) \times 0 = 0$ . The weighted sum is $1.5 + 0 = 1.5$ , and adding the bias gives $z = 1.5 + 0.25 = 1.75$ . The total is positive, so the fan should run.
Apply ReLU to the positive total. The comparison is $\max(0, 1.75) = 1.75$ . A positive total passes through unchanged, so the output is 1.75 on the made-up scale, a strength that rises as the room gets warmer.

Answer: With the window open, the output is 0, so the fan stays off, because the negative total was switched to zero. With the window shut, the output is 1.75, so the fan runs. The same ReLU rule either passes a positive total through unchanged or turns a negative total into zero.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Thinking ReLU is a curve that dips below zero. The output is never negative: every negative input becomes exactly 0.
Applying ReLU to the inputs instead of to the weighted sum. The activation goes on the neuron’s total, after the weights and bias have been added.
Expecting a neuron to recover on its own after it outputs 0. If its input stays negative, its gradient is zero and its weights stop changing, so it can stop learning.
Using ReLU as the output of a yes-or-no classifier. Its output is not a probability, so use a sigmoid or a softmax at the output instead.

Where you will meet it

Hidden layers of convolutional networks for image recognition, where it is a common default.
Hidden layers of fully connected networks for regression and classification tasks.
Output layers of networks that must predict a value that is never negative, such as a count or a length.
Introductory deep learning courses, where it is usually the first activation function students meet after the sigmoid.

Questions people ask

Why not use a straight line instead?

A stack of straight-line layers is still one straight line overall. The bend that ReLU adds is what lets stacked layers represent curved patterns.

What is the derivative of ReLU?

It is 1 for positive inputs and 0 for negative inputs. The point exactly at zero is handled by convention in code, usually as 0.

Why is ReLU so popular?

It is fast to compute, it does not flatten out large values the way a sigmoid does, and it trains well in deep networks. Those three points together made it a common default for hidden layers.

Tanh Activation: Tanh squeezes any number into a value between minus 1 and plus 1, with zero staying at zero, so a negative input always gives a negative output.
Sigmoid Function: The sigmoid takes any number, however big or small, and squeezes it into a value between 0 and 1 that can be read as a chance, like 0.8 meaning 80 percent.
Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds the results together with one extra number called the bias, and then uses a simple rule to decide how strong its output should be.
Backpropagation and the Chain Rule: Backpropagation finds how much each weight affects the error by multiplying the small slopes along the chain from that weight to the loss, one link at a time.