Adam Optimizer

In one sentence: Adam keeps a running average of each setting’s slope and of how large those slopes have been, then uses both to choose a step size for every setting separately.

$\theta_t = \theta_{t-1} - \eta \, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}$

Read it out loud: “theta sub t equals theta sub t minus 1, minus eta times m-hat sub t, divided by the square root of v-hat sub t, plus epsilon. Here theta is the settings of the model, and the subscript t counts steps. Eta, said AY-tuh, is the learning rate. M-hat is the corrected average of the slope, and v-hat is the corrected average of the squared slope. Epsilon, said EP-sih-lon, is a tiny safety number.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Settings, slopes and subscripts: The settings of the model are written $\theta$ , the Greek letter theta, said THAY-tuh. The subscript $t$ counts steps, starting from 1, so $\theta_t$ is the settings after step $t$ and $\theta_{t-1}$ is the settings one step earlier. The gradient $g_t$ is the slope of the loss at the settings before step $t$ . It says which way the loss rises fastest.
Running averages and decay rates: A running average is an average that is updated after every step, so each new value enters a little and the old values fade. The fraction of the old average that is kept each step is called the decay rate. Adam uses two decay rates, $\beta_1$ (beta one, usually 0.9) for the gradient average and $\beta_2$ (beta two, usually 0.999) for the squared-gradient average. Because $\beta_2$ is closer to 1, its average changes more slowly.
Hats, squares and the bias correction, like $\hat{m}_t$: A small hat over a letter, as in $\hat{m}_t$ (read m-hat), marks a corrected version of a value. The correction adjusts the averages for their slow start. A square such as $g_t^{2}$ means $g_t$ times $g_t$ . It is never negative, so the average of squares measures how large the slopes have been, while ignoring whether they went up or down.
Square roots, $\sqrt{x}$: The square root $\sqrt{x}$ is the number that, multiplied by itself, gives $x$ . For example $\sqrt{36} = 6$ , because $6 \times 6 = 36$ . Taking a square root turns a squared quantity back into the same size as the original slopes. That is why Adam divides by the square root of the averaged squares.
Powers of ten, like $10^{-8}$: Very small numbers are written with a power of ten. The exponent says how many times to divide by 10. So $10^{-8} = 1 / 10^{8}$ , which is 0.00000001: a decimal point, then seven zeros, then a 1. Adam’s safety number $\epsilon$ is set to $10^{-8}$ , so it is tiny, but not zero. Writing it this way keeps the number short and easy to read.
Powers of a decay rate, like $\beta_1^{t}$: The power $\beta_1^{t}$ means $\beta_1$ multiplied by itself $t$ times. With $\beta_1 = 0.9$ , after one step it is 0.9, after two steps $0.9 \times 0.9 = 0.81$ , and after three steps $0.729$ . These powers shrink toward zero as $t$ grows, and they are used in the bias correction.

What each symbol means

Symbol	Meaning
$\theta_t$	The settings of the model after step $t$ . Step counting starts at 1 for the first update. The settings are a list of numbers, and in this toy example they are unitless.
$\theta_{t-1}$	The settings before step $t$ , one update earlier. The gradient is measured at this point, so it is the place where the slope is read.
$\eta$	The learning rate, the base size of each step. It is a small positive number, and the Adam paper suggests 0.001 as a starting value. It has no unit in this formula, but it sets how far the settings move per step.
$\hat{m}_t$	The bias-corrected average of the gradient, read m-hat sub t. It tracks the direction the slope has been pointing recently. Its unit is the same as the gradient’s unit.
$\hat{v}_t$	The bias-corrected average of the squared gradient, read v-hat sub t. Squares are never negative, so this number measures how large the slopes have been. Its unit is the square of the gradient’s unit.
$\sqrt{\hat{v}_t}$	The square root of the averaged squared gradient. It gives the typical size of the recent slopes in the same unit as the gradient, so dividing by it makes each step depend on how steady the slopes are.
$\epsilon$	A tiny positive number, usually $10^{-8}$ , added to the bottom of the fraction so the step never divides by zero. It is unitless, and it has almost no effect unless the typical slope size is extremely small.
$g_t$	The gradient at step $t$ , the slope of the loss at the settings before that step. It has the unit of the loss divided by the unit of the settings. It is the raw input to both running averages.
$\beta_1$	The decay rate for the gradient average, the Greek letter beta with subscript 1. Usually 0.9, so the old average keeps 90 percent of its weight each step.
$\beta_2$	The decay rate for the squared-gradient average, the Greek letter beta with subscript 2. Usually 0.999, so the old squared average keeps 99.9 percent of its weight each step, which makes it change slowly.

The formula piece by piece

$\hat{m}_t$: The averaged slope, corrected for the early steps. It points in the direction the loss has been falling, averaged over recent steps, so one noisy slope cannot decide the whole move.
$\sqrt{\hat{v}_t} + \epsilon$: The typical size of recent slopes, with the tiny safety number $\epsilon$ added so that nothing divides by zero. Taking the square root returns the squared average to the original size of the slopes.
$\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$: The direction divided by the typical size. When the slopes have been steady, this ratio is close to plus or minus 1, so each step is about the learning rate. When the slopes flip sign, the average shrinks and the step gets smaller.
$\eta \, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$: The learning rate times that ratio: the size of one step, in the units of the settings. Each setting gets its own ratio, so each setting moves at its own pace.
$\theta_t = \theta_{t-1} - \eta \, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}$: The whole rule in one sentence: the new settings equal the old settings minus the learning rate times the averaged slope divided by its typical size, with a tiny safety number to keep the division safe.

What the formula is doing, and why

What goes in and what comes out. Adam takes the current slope for each setting and returns a new value for that setting. Inside, it keeps two running averages for every setting. The first, $m_t$ , is an average of the slopes $g_t$ , so it shows which way the slope has been pointing. The second, $v_t$ , is an average of the squared slopes, so it shows how large the slopes have been. They update as $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ and $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^{2}$ . Each new slope takes a small share, and the old average keeps most of its weight.

Why correct the early steps? Both averages start at zero, so the first few values are too small, because they have not yet collected enough slopes. The bias correction fixes this: $\hat m_t = m_t / (1 - \beta_1^{t})$ and $\hat v_t = v_t / (1 - \beta_2^{t})$ . The denominators are small early on, which makes the division bigger, and they approach 1 as $t$ grows. So the correction matters mostly for the first steps.

Why divide by the typical slope size? The update divides the averaged slope $\hat m_t$ by the square root of the averaged squared slope, $\sqrt{\hat v_t}$ . Dividing by the typical size means that each setting moves by roughly $\eta$ , whatever the raw size of its slopes was. Settings with steady, large slopes do not take huge steps, and settings with small slopes still move. The tiny $\epsilon$ only keeps the division safe when the typical slope size is almost zero.

Defaults and their limits. The Adam paper by Kingma and Ba reports $\beta_1 = 0.9$ , $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$ , with the learning rate $\eta = 0.001$ , as settings that worked well on the problems it tested. Many libraries use the same defaults. They are a starting point, not a guarantee of the best result on a new problem.

An everyday picture

Picture a driver who remembers two things about the road: which way it has been bending, and how bumpy the ride has been. On a smooth stretch the driver steers firmly. On a rough, jerky stretch the driver takes smaller, careful steps. Adam gives each setting in the model its own memory of how bumpy its road has been.

Worked example: Two steps of Adam on one setting

One setting, $\theta$ , has a bowl-shaped loss $L(\theta) = (\theta - 3)^{2}$ , with its lowest point at 3. Its slope is $g = 2(\theta - 3)$ . We start at $\theta_0 = 0$ with both averages at zero, $m_0 = 0$ and $v_0 = 0$ . We use the defaults $\beta_1 = 0.9$ , $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$ . For easy numbers, the learning rate is $\eta = 0.1$ . All values are unitless.

Step 1: find the slope. At $\theta_0 = 0$ the slope is $g_1 = 2(0 - 3) = -6$ . The slope is negative, which means the loss falls as the setting goes up, so the step should move the setting upward.
Step 1: update the two averages. The gradient average becomes $m_1 = 0.9 \times 0 + 0.1 \times (-6) = -0.6$ . The squared average becomes $v_1 = 0.999 \times 0 + 0.001 \times 36 = 0.036$ . Each average takes only a small share of the new value, because the decay rates keep most of the old average.
Step 1: correct the early averages. Both averages are still small, because they started at zero. We divide by $1 - \beta_1^{1} = 0.1$ and by $1 - \beta_2^{1} = 0.001$ . That gives $\hat m_1 = -0.6 / 0.1 = -6$ and $\hat v_1 = 0.036 / 0.001 = 36$ , which match the slope we actually measured.
Step 1: move the setting. The ratio $\hat m_1 / \sqrt{\hat v_1} = -6 / 6 = -1$ . So the step is the learning rate times that ratio: $\theta_1 = 0 - 0.1 \times (-1) = 0.1$ , approximately. The first step moves by exactly the learning rate, which is what this ratio is designed to do when the slopes are steady.
Step 2: find the new slope and update the averages. At $\theta_1 = 0.1$ the slope is $g_2 = 2(0.1 - 3) = -5.8$ . The gradient average is $m_2 = 0.9 \times (-0.6) + 0.1 \times (-5.8) = -1.12$ . The squared average is $v_2 = 0.999 \times 0.036 + 0.001 \times 33.64 = 0.069604$ .
Step 2: correct the averages and move. Dividing by $1 - 0.9^{2} = 0.19$ gives $\hat m_2 \approx -5.8947$ . Dividing by $1 - 0.999^{2} \approx 0.001999$ gives $\hat v_2 \approx 34.8194$ . The ratio is $-5.8947 / \sqrt{34.8194} \approx -0.9989$ , so $\theta_2 = 0.1 - 0.1 \times (-0.9989) \approx 0.1999$ . The second step is again close to the learning rate, even though the raw slope was 5.8.

Answer: After two steps the setting is $\theta_2 \approx 0.1999$ . Both steps are almost exactly the learning rate of 0.1, even though the raw slope was 6 at step 1 and 5.8 at step 2. Adam sets the step size from the ratio of the averaged slope to its typical size, not from the raw slope alone.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Skipping the bias correction. Because $m_t$ and $v_t$ start at zero, the early averages are too small, and the first few steps come out several times too big for these defaults.
Expecting every step to be exactly $\eta$ . The step is about $\eta$ only when the slopes are steady. When the slopes flip sign, $\hat m_t$ shrinks compared with $\sqrt{\hat v_t}$ , and the step gets smaller.
Setting $\epsilon$ too large. When $\sqrt{\hat v_t}$ is small, a big $\epsilon$ dominates the bottom of the fraction and quietly shrinks every step, most of all for settings with small slopes.
Keeping the default learning rate without checking it. The defaults come from the paper’s test problems and are a starting point, not a rule for every model.

Where you will meet it

Training large text models, such as transformers, where Adam or a close variant called AdamW is a common choice.
Training image and speech models, including many deep networks for recognizing pictures and sounds.
Deep learning libraries, where Adam is one of the built-in optimizers, the rules that update the settings during training.
Fine-tuning pretrained models, which means adjusting an existing model on new data, often with a small learning rate and Adam together.

Questions people ask

Why is it called Adam?

The name comes from adaptive moment estimation. The first moment is the averaged slope and the second moment is the averaged squared slope, so the name describes the two running averages that Adam keeps.

Does Adam always beat plain gradient descent?

No. Adam often trains quickly with little tuning, but on some problems gradient descent with momentum on small batches reaches a better final result. Trying both is sensible when the model matters.

What is the difference between m and v?

$m_t$ is the average of the slope itself, which carries the direction. $v_t$ is the average of the squared slope, which carries the size. Adam uses both to decide how far each setting moves.

Do I need to change beta 1 and beta 2?

Usually not at first. The defaults 0.9 and 0.999 are the starting point in most libraries.

Momentum Optimizer: Momentum lets gradient descent remember its recent steps, so it speeds up where the slope keeps pointing the same way and wobbles less where the slope keeps flipping.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.
Backpropagation and the Chain Rule: Backpropagation finds how much each weight affects the error by multiplying the small slopes along the chain from that weight to the loss, one link at a time.