# ReLU Activation

> ReLU keeps positive numbers unchanged and replaces every negative number with zero, like a switch that is on when the input is positive and off otherwise.

Page: https://formulabro.yunusemrevurgun.com/ai-ml/relu-activation/  
Category: AI & Machine Learning  
Published: 2026-10-22  
Updated: 2026-10-22

## Formula

$$
\text{ReLU}(z) = \max(0, z)
$$

Read it out loud: "ReLU of z equals the larger of zero and z. ReLU is short for rectified linear unit. The letter z is the input number, which is usually a weighted sum plus a bias, and max means take the bigger of the two numbers."

## Before you start

- **Positive, negative and zero on the number line**: Numbers to the right of zero on the number line are positive, and numbers to the left are negative. Zero sits between them and is neither. A negative number is a value below zero, such as a temperature of minus 5 degrees Celsius. Multiplying two negative numbers gives a positive number, and adding a negative number is the same as subtracting its size.
- **The maximum $\max(a, b)$**: $\max(a, b)$ means 'the bigger of the two numbers $a$ and $b$'. So $\max(0, 4) = 4$, because 4 is bigger than 0, and $\max(0, -3) = 0$, because 0 is bigger than $-3$. The notation picks one of the two numbers and throws the other away, which is exactly what the ReLU rule does to a negative input.
- **An activation function**: An **activation function** is a rule that a neuron applies to its total before passing the result on. It decides how strong the output is. Without a rule like this, a stack of neuron layers would combine into one straight-line formula. The activation function adds a bend, which is what lets the network represent curved patterns.
- **Straight lines, slope and linear**: A straight line has a constant **slope**, the amount it rises for each step to the right. The line $y = z$ has slope 1, because each step to the right rises by one step up. A rule is called **linear** when it is a straight line in this sense. ReLU is not linear overall, because it is a straight line with slope 1 on the positive side and a flat line at 0 on the negative side, which makes a bend.
- **Gradient, the slope used for learning**: The **gradient** of a function is how much its output changes when its input changes a tiny bit. During training, gradients tell the network which way to adjust its weights. For ReLU, the gradient is 1 on the positive side, because the output follows the input, and 0 on the negative side, because the output stays fixed at zero no matter how the input moves.

More notation help: https://formulabro.yunusemrevurgun.com/math-basics/

## What each symbol means

- $\text{ReLU}(z)$: The **rectified linear unit output**: the value the neuron passes on to the next layer. It is never negative, because negative inputs become 0. It has no unit in most networks, and it is a plain number like the input.
- $z$: The **input** to the activation, usually the weighted sum of a neuron's inputs plus its bias. It can be any real number, positive, negative or zero. In the example it is a made-up score that says how much the fan should run, and it has no unit.
- $\max(0, z)$: The larger of the two numbers 0 and $z$. If $z$ is positive, the result is $z$ itself, passed on unchanged. If $z$ is negative, the result is 0. This is the entire rule, written as one comparison.

## The formula piece by piece

- $z = w_1 x_1 + w_2 x_2 + b$: Start with the neuron's weighted sum. Each input is multiplied by its weight, the products are added, and the bias is added last. This total $z$ is the number ReLU will read, and it can be positive or negative depending on the inputs.
- $\max(0, z)$: Compare the total with 0 and keep the bigger one. A positive total is kept exactly as it is. A negative total loses the comparison and becomes 0, so the neuron outputs nothing for that input.
- $\text{ReLU}(z) = \max(0, z)$: The whole formula in one sentence: the ReLU output of $z$ equals the larger of zero and $z$, so positive values pass through unchanged and negative values become zero.

## What the formula is doing, and why

An activation function sits at the output of a neuron and decides what gets passed on. A neuron first computes a weighted sum $z$ of its inputs, then applies the activation. Without a non-linear activation, a stack of layers collapses into one big linear formula, and that formula cannot learn curved patterns.

**What does ReLU do?** ReLU, short for rectified linear unit, is the simplest useful choice, and it has two cases. If $z$ is positive, the output equals $z$, a straight line with slope 1. If $z$ is zero or negative, the output is 0. The output never goes below zero. The bend at zero is the only non-straight part, and that bend is enough to let stacked layers learn curved patterns.

**Why is it popular?** The main advantage is cheap computation. Producing the output takes one comparison, with no exponentials. For positive inputs the slope is always 1, so the learning signal does not shrink there the way it can with curves such as the sigmoid, which flatten out at both ends. That is a large part of why deep networks train well with it.

**What is the known drawback?** The dying ReLU problem. A neuron whose input stays negative outputs 0 and receives a gradient of 0, so its weights stop changing, and it can stop learning for good. Variants such as the leaky ReLU keep a small slope on the negative side to avoid this.

## An everyday picture

Think of a one-way valve on a water pipe. Water pushing forward flows through unchanged, so a bigger push gives a bigger flow. Water pushing backward meets a closed valve and stops at zero. The neuron lets through only the part of the signal that points in the forward direction.

## Worked example: A smart fan that depends on the window

A smart fan uses one neuron to decide how hard to run. Its inputs are $x_1 = 3$, the number of degrees the room is above a comfortable temperature, and $x_2$, which is 1 when a window is open and 0 when it is shut. The weights are $w_1 = 0.5$ per degree and $w_2 = -2$ for the open window, because open windows cool the room. The bias is $b = 0.25$. The fan's strength is on a made-up scale with no unit. We find the output with the window open, then with the window shut.

1. **Compute the weighted sum with the window open.** Here $x_2 = 1$. The first input contributes $0.5 \times 3 = 1.5$, and the open window contributes $(-2) \times 1 = -2$. Those two products are added, which gives $1.5 - 2 = -0.5$, so far the signal points toward turning the fan off.
2. **Add the bias.** The bias $0.25$ is added to the weighted sum, so $z = -0.5 + 0.25 = -0.25$. The total is negative, which means the room would be cooled by the open window, so the fan should not work hard.
3. **Apply ReLU to the negative total.** ReLU compares the total with zero, $\max(0, -0.25) = 0$. The negative total loses, so the output is 0. The fan is off, and the neuron has switched itself off for this input.
4. **Now shut the window.** With the window closed, $x_2 = 0$, so the window term is $(-2) \times 0 = 0$. The weighted sum is $1.5 + 0 = 1.5$, and adding the bias gives $z = 1.5 + 0.25 = 1.75$. The total is positive, so the fan should run.
5. **Apply ReLU to the positive total.** The comparison is $\max(0, 1.75) = 1.75$. A positive total passes through unchanged, so the output is 1.75 on the made-up scale, a strength that rises as the room gets warmer.

**Answer:** With the window open, the output is **0**, so the fan stays off, because the negative total was switched to zero. With the window shut, the output is **1.75**, so the fan runs. The same ReLU rule either passes a positive total through unchanged or turns a negative total into zero.

## Common mistakes

- Thinking ReLU is a curve that dips below zero. The output is never negative: every negative input becomes exactly 0.
- Applying ReLU to the inputs instead of to the weighted sum. The activation goes on the neuron's total, after the weights and bias have been added.
- Expecting a neuron to recover on its own after it outputs 0. If its input stays negative, its gradient is zero and its weights stop changing, so it can stop learning.
- Using ReLU as the output of a yes-or-no classifier. Its output is not a probability, so use a sigmoid or a softmax at the output instead.

## Where you will meet it

- Hidden layers of convolutional networks for image recognition, where it is a common default.
- Hidden layers of fully connected networks for regression and classification tasks.
- Output layers of networks that must predict a value that is never negative, such as a count or a length.
- Introductory deep learning courses, where it is usually the first activation function students meet after the sigmoid.

## Questions people ask

### Why not use a straight line instead?

A stack of straight-line layers is still one straight line overall. The bend that ReLU adds is what lets stacked layers represent curved patterns.

### What is the derivative of ReLU?

It is 1 for positive inputs and 0 for negative inputs. The point exactly at zero is handled by convention in code, usually as 0.

### Why is ReLU so popular?

It is fast to compute, it does not flatten out large values the way a sigmoid does, and it trains well in deep networks. Those three points together made it a common default for hidden layers.

## Related formulas

- [Tanh Activation](https://formulabro.yunusemrevurgun.com/ai-ml/tanh-activation/)
- [Sigmoid Function](https://formulabro.yunusemrevurgun.com/ai-ml/sigmoid-function/)
- [Artificial Neuron (Weighted Sum)](https://formulabro.yunusemrevurgun.com/ai-ml/artificial-neuron/)
- [Backpropagation and the Chain Rule](https://formulabro.yunusemrevurgun.com/ai-ml/backpropagation-chain-rule/)

## Sources

- [Rectifier (neural networks), Wikipedia](https://en.wikipedia.org/wiki/Rectifier_%28neural_networks%29)
- [Activation function, Wikipedia](https://en.wikipedia.org/wiki/Activation_function)

From Formula Bro (https://formulabro.yunusemrevurgun.com/), a side project developed by yunusemrevurgun.com.