Dropout

In one sentence: Dropout randomly turns off some of a network’s neurons during each practice round and boosts the ones left on, so the network cannot lean on any single neuron.

Read it out loud: “h-tilde sub i equals m sub i times h sub i, all divided by one minus p. Here h sub i is the output of neuron number i, and m sub i is a random mask that is 1 when the neuron is kept and 0 when it is dropped. The mask follows a Bernoulli draw with keep probability one minus p, which means it is 1 with that probability and 0 otherwise. The letter p is the drop probability, a number between 0 and 1.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Neuron and activation

A neuron in a neural network is a small calculator. It takes in several numbers, combines them with learned weights, and produces one output number. That output is called its activation, written $h_i$ here. A network has thousands or millions of neurons arranged in layers, and the activations of one layer become the inputs of the next.

Training pass and prediction

A training pass is one round in which the network sees examples and adjusts its weights. Prediction, also called inference, is when the finished network answers a new question. Dropout works only during training, and prediction uses the whole network with no neurons switched off.

Probability as a number from 0 to 1

A probability is a number from 0 to 1 that says how likely something is: 0 means it never happens, 1 means it always happens, and 0.5 means a fair coin, heads half the time. So $p = 0.5$ means each neuron has a 50% chance of being dropped on a pass, and $1 - p = 0.5$ is the chance it is kept.

Bernoulli draw and the mask $m_i$

A Bernoulli draw is one random yes-or-no trial. It gives 1 with some probability and 0 otherwise, like a weighted coin. The name comes from the Swiss mathematician Jacob Bernoulli. The mask $m_i$ is one such draw for one neuron. Each neuron gets a fresh draw on every training pass, so the masks change from pass to pass.

Long-run average, or expected value

An expected value is the long-run average of a random result. If you repeat a random choice many times and average the outcomes, the average settles near the expected value. The scaling step in dropout is designed so that the expected value of each neuron’s output equals the output it would have had with every neuron on.

Tilde over a letter, like $\tilde h_i$

A little wave over a letter, read tilde, marks a changed version of the same quantity. $\tilde h_i$ (read h-tilde sub i) is the activation after dropout, while $h_i$ is the activation before dropout. The wave is only a label, so $\tilde h_i$ is a number of the same kind as $h_i$, changed only by the mask and the scaling.

Overfitting and regularization

Overfitting means a model memorizes its training examples so well that it does poorly on new examples. A regularizer is a method that discourages this. Dropout is a regularizer: by practicing with random parts missing, the network is pushed to learn patterns that are spread across many neurons, not stored in a few.

What each symbol means

The formula piece by piece

$h_i$

The starting point is the activation of one neuron: the number it produces before dropout. Every other piece of the formula acts on this number, one neuron at a time, so the calculation is repeated for each neuron in the layer.

$m_i h_i$

Multiply the activation by its mask. If the mask is 1, the activation passes through unchanged. If the mask is 0, the product is 0 however large the activation is, so the neuron is effectively removed for this training pass.

$\frac{1}{1 - p}$

The scale factor. Dividing by the keep probability makes the surviving neurons stronger. Because about a fraction $1 - p$ of the neurons survive, this boost keeps the total signal about the same on average, even though some neurons are missing.

$\tilde h_i = \frac{m_i \, h_i}{1 - p} \qquad m_i \sim \text{Bernoulli}(1 - p)$

On each training pass every neuron is kept or switched off at random, and the kept outputs are scaled up by one over one minus p, so the average signal stays the same.

$\tilde h_i = h_i$

The same law at prediction time, when dropout is turned off. No mask is drawn and no scaling is applied, so every neuron keeps its full activation and the layer passes its values on.

What the formula is doing, and why

A network with many weights can memorize its training examples instead of learning general patterns. Dropout is a training method that fights this. On each training pass, every neuron is switched off at random with probability $p$. The neurons that remain must carry the signal, so no single neuron can become essential. If one neuron happens to be a strong shortcut, dropout forces the others to learn to carry the signal as well.

Why multiply by a mask? Multiplying by $m_i$ is the simplest way to switch a neuron off. A 0 turns the activation into 0, and a 1 leaves it alone. Each neuron is kept with probability $1 - p$, so on average a fraction $1 - p$ of the activations survive the pass.

Why divide by $1 - p$? Without this step, the surviving neurons would deliver less signal on average than they do when all neurons are on, and the next layer would see inputs that are too weak during training. Dividing by $1 - p$ scales up the kept activations to make up for the ones removed. When $p = 0.5$ the kept values are doubled. This version is called inverted dropout, and most libraries use it. Over many random masks, the average value of $\tilde h_i$ equals $h_i$.

Why no dropout at prediction time? When you use the finished network to answer a question, you want the same answer every time. Random dropping would change the answer from run to run, so dropout is switched off. Every neuron stays on, and the activations pass through unchanged. Because training already scaled the kept values, the average signal matches. The original 2014 paper applied the scaling to the weights at prediction time instead, which gives the same average signal.

When does it help? Dropout is useful when a model overfits, meaning it does much better on its training examples than on new ones. A small model that already underfits gains little from dropout, because switching neurons off only removes signal it needs. The right amount of dropout depends on the model and the data, and it is usually tuned on separate validation examples.

An everyday picture

Think of a sports team that practices with a few players sitting out each drill, chosen at random. Every player has to learn to play well with different teammates, so no one player becomes a crutch. On game day everyone plays, and the team is stronger for having practiced without any single star. In the same way, dropout makes a network practice without any single neuron, so its learned skills are shared.

Worked example: Four neurons and one training pass

A layer has four neurons. Their activations before dropout are $h = (0.5,\ 2.0,\ 1.0,\ 4.0)$, plain numbers with no unit. The drop probability is $p = 0.5$, so each neuron is kept with a 50% chance, like a fair coin. The random draw for this pass gives the mask $m = (1, 0, 1, 0)$: neurons 1 and 3 are kept, and neurons 2 and 4 are dropped.

1. Read the mask. The mask says what happens to each neuron on this pass: 1 means keep it and 0 means switch it off. Neurons 1 and 3 pass through, and neurons 2 and 4 are set to zero. Each neuron has its own yes-or-no draw, so the four decisions are separate from one another.
2. Find the scale factor. The keep probability is $1 - p = 1 - 0.5 = 0.5$. We divide by this number, so the scale factor is $\frac{1}{0.5} = 2$. Dividing by one half is the same as doubling, so every kept activation will be doubled before the next layer sees it.
3. Neuron 1 is kept. Its mask is 1, so we multiply its activation by 1 and then divide by 0.5: $\tilde h_1 = \frac{1 \times 0.5}{0.5} = 1.0$. The value 0.5 becomes 1.0. The multiplication by 1 leaves it alone, and the division doubles it.
4. Neuron 2 is dropped. Its mask is 0, so the product is zero however large the activation is: $\tilde h_2 = \frac{0 \times 2.0}{0.5} = 0$. The activation 2.0 is removed completely for this pass, so the next layer receives nothing from this neuron.
5. Neuron 3 is kept. Its mask is 1, so $\tilde h_3 = \frac{1 \times 1.0}{0.5} = 2.0$. The value 1.0 is doubled to 2.0, which makes up for the neurons that were switched off and keeps the layer’s total signal in balance.
6. Neuron 4 is dropped. Its mask is 0, so $\tilde h_4 = \frac{0 \times 4.0}{0.5} = 0$. The large value 4.0 is removed, and the layer’s output for this training pass is $(1.0,\ 0,\ 2.0,\ 0)$.
7. Check the average over many passes. A neuron is kept half the time, so its average output is half its kept value plus half of zero. Neuron 1 averages $0.5 \times 1.0 + 0.5 \times 0 = 0.5$, which equals its original value. Neuron 2 is kept in some passes at $\frac{2.0}{0.5} = 4.0$, so it averages $0.5 \times 4.0 + 0.5 \times 0 = 2.0$, its original value again.
8. Prediction time. When the finished network answers a question, no mask is drawn and no scaling is applied. The layer outputs $(0.5,\ 2.0,\ 1.0,\ 4.0)$ unchanged, so the same input always gives the same answer, which is what we want when the network is used.

Answer: During this training pass the layer outputs $(1.0,\ 0,\ 2.0,\ 0)$: two neurons were switched off and the two kept ones were doubled. Over many random passes each neuron’s average output equals its original value, so the layer’s average signal is unchanged. At prediction time the layer outputs the original values, because dropout is turned off.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

• Dropping neurons at prediction time. Dropout is a training tool. At prediction time the full network is used, and keeping the random masks would make the answers change from run to run.
• Forgetting the $1 / (1 - p)$ scaling in inverted dropout. Then the signal is too weak during training, and the layer’s outputs no longer match what the next layer expects at prediction time.
• Setting $p$ too high, such as 0.9. Most neurons are switched off on every pass, so the network learns slowly and can underfit the data.
• Expecting dropout to add capacity. It is a regularizer that helps when a model overfits. It does not make a small model bigger or smarter.

Where you will meet it

• Fully connected layers in image and text classifiers, where the 2014 paper by Srivastava and colleagues showed it reduces overfitting.
• Transformer models, which apply dropout to attention weights and to the outputs of their sublayers during training.
• Recurrent networks and other models trained on limited data, where overfitting is a serious risk.
• Monte Carlo dropout, where dropout stays on at prediction time, so that repeated predictions with different masks show how uncertain the model is.

Questions people ask

Is p the keep probability or the drop probability?

Here $p$ is the drop probability, and the keep probability is $1 - p$. Some papers, including the original 2014 paper, use the letter $p$ for the keep probability instead, so check each source’s definition before you copy a formula.

What is a good value of p?

Many classifiers use values between 0.1 and 0.5. The best value depends on the model and the data, so it is usually chosen by trying a few values on separate validation examples.

Why does dropout make a network more robust?

Each neuron has to be useful alongside a different set of partners on each pass. No single neuron can be relied on, so the features the network learns are spread across many neurons, and the network copes better when new data is a little different from the training data.

Related formulas

• Batch Normalization: Batch normalization rescales a group of values so they average out to zero with a steady spread, then lets the network learn its own final scale and shift.
• Artificial Neuron (Weighted Sum): An artificial neuron multiplies each input by a weight, adds the results together with one extra number called the bias, and then uses a simple rule to decide how strong its output should be.
• Cross-Entropy Loss: Cross-entropy loss gives a small penalty when the model gave the right answer a high probability, and a big penalty when it gave the right answer a small probability.

Sources

1. Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Srivastava et al., JMLR 15 (2014)
2. Dilution (neural networks), Wikipedia