Discounted Return

Q: What does a discount factor of 0 mean?

The agent cares only about the next reward. Every later term is multiplied by $0^{k} = 0$ for $k \geq 1$, so the return is $r_{t+1}$ alone.

Q: Why must $\gamma$ stay below 1?

With $\gamma < 1$ and bounded rewards, the sum has a finite total. With $\gamma = 1$ in a task that never ends, the return can grow without limit.

Q: Is the return the same as the reward?

No. The reward is what the agent gets from one step. The return adds up all the rewards from time $t$ onward, each one discounted.

Q: Who chooses $\gamma$?

The designer of the agent. It is a setting, like a learning rate, and it changes what the agent cares about.

In one sentence: The discounted return is the total reward an agent can expect from now on, where rewards that arrive later count for a little less.

$G_t = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1}$

Read it out loud: “G sub t equals the sum, for k from 0 to infinity, of gamma raised to the power k, times the reward r sub t plus k plus 1. G sub t is the return at time t, which is the total from that moment onward. The capital sigma means add up, and the infinity sign means the sum has no last term. Gamma, said GAM-uh, is the discount factor, a number from 0 up to but not including 1. The letter k counts how many steps after the first reward, and t is the current time step.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Time steps, rewards and the return: A time step is one round of acting: the agent acts, and the world answers. A reward, written $r$ , is the number of points received at one step. The return, written $G$ , is the total of all the rewards from some moment onward. Reinforcement learning is the field where a program learns to act by trying things and collecting rewards.
Subscripts for time, like $r_{t+k+1}$: The letter $t$ stands for the current time step, and $k$ counts steps after it. The subscript $t+k+1$ therefore names the step that is $k+1$ steps after time $t$ . So $r_{t+1}$ is the reward at the very next step, and $r_{t+3}$ is the reward three steps after time $t$ . Subscripts only label which reward is meant.
Powers, like $\gamma^{k}$: A power, written with a raised number, means multiplying a number by itself that many times. So $\gamma^{2} = \gamma \times \gamma$ , and $\gamma^{0} = 1$ for any $\gamma$ . With $\gamma = 0.9$ , we get $0.9^{2} = 0.81$ and $0.9^{3} = 0.729$ . Because $\gamma$ is below 1, each higher power is smaller than the one before it.
The discount factor, $\gamma$: The Greek letter $\gamma$ , called gamma and said GAM-uh, is the discount factor. It is a number from 0 up to but not including 1. It sets how much a reward loses value for each step it is delayed. A discount of 0.9 means a reward one step later counts for 90 percent of its face value.
The sum sign and the infinity sign: The capital sigma, $\sum$ , means add up. The infinity sign, $\infty$ , means without end, so $\sum_{k=0}^{\infty}$ adds the terms for $k = 0, 1, 2$ and so on forever. A sum like this has a finite total when its terms shrink fast enough. The discount does that when $\gamma < 1$ and the rewards never grow past some fixed size.

What each symbol means

Symbol	Meaning
$G_t$	The return at time $t$ : the total discounted reward from that moment onward. It is measured in reward points, with no unit.
$t$	The current time step, the moment from which the return is counted. It is a count of steps, with no unit.
$\sum_{k=0}^{\infty}$	The add-up sign over every $k$ from 0 to infinity. In principle the sum has no last term, but its later terms become tiny, so in practice it stops when the rewards run out.
$k$	How many steps after the first reward a discounted reward arrives. $k = 0$ is the very first reward in the sum, and $k = 1$ is the one after it. It is a count, with no unit.
$\gamma$	The discount factor, with $0 \le \gamma < 1$ . It sets how much less a reward counts for each step it is delayed. It has no unit.
$r_{t+k+1}$	The reward received at the step that is $k+1$ steps after time $t$ . It is measured in reward points.

The formula piece by piece

$r_{t+k+1}$: One reward, received $k + 1$ steps after time $t$ . For $k = 0$ it is the reward for the very next step, which is the first reward the sum counts.
$\gamma^{k}$: The discount weight for a reward that comes $k$ steps after the first one. It is 1 for the first reward, and each extra step multiplies the weight by $\gamma$ once more.
$\gamma^{k} r_{t+k+1}$: One discounted reward: the reward multiplied by its weight. The later a reward arrives, the smaller its weight, so the less it adds to the return.
$G_t = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1}$: The return at time $t$ equals the sum of all the future rewards, each one multiplied by $\gamma$ raised to the number of steps after the first reward.
$G_t = r_{t+1} + \gamma G_{t+1}$: The same return in a shortcut form. It says the return now equals the next reward, plus $\gamma$ times the return one step later. Learning methods use this form to update their estimates one step at a time.

What the formula is doing, and why

The input is the stream of rewards after time $t$ , and a discount factor $\gamma$ . The output is one number, the return $G_t$ : the total reward from now on. The agent wants this total to be as large as possible. The formula adds the rewards in order, first $r_{t+1}$ , then $r_{t+2}$ , and so on, each one weighted by how far away it arrives.

Why multiply by $\gamma^{k}$ ? Rewards that come later are less certain, and they matter less for the decision made now. Each extra step multiplies the weight by $\gamma$ once more. With $\gamma = 0.9$ , a reward two steps after the first counts for $0.9^2 = 0.81$ of its face value, and a reward three steps after counts for $0.9^3 = 0.729$ . Rewards far in the future count for less and less.

Why must $\gamma$ stay below 1? If every reward has size at most $R$ , the sum is at most $R + \gamma R + \gamma^2 R + \cdots$ . That is a geometric series, a sum where each term is the last one times a fixed number. Its total is $\frac{R}{1 - \gamma}$ . For example, with $\gamma = 0.5$ the sum $1 + 0.5 + 0.25 + \cdots$ gets close to 2 but never passes it. So with $\gamma < 1$ the return stays a finite number, which makes returns comparable.

The shortcut. The return from time $t$ is the next reward plus the return from the next step, discounted once: $G_t = r_{t+1} + \gamma G_{t+1}$ . This works because the rest of the sum, starting one step later, is the same kind of sum with one extra factor of $\gamma$ at the front. Learning methods use this shortcut to update their estimates one step at a time.

When it applies. The formula fits any task in which the agent earns rewards over time. The designer chooses $\gamma$ . A value close to 1 values the long run, and a smaller value values quick rewards more. For a task with no end, $\gamma$ must stay below 1 so that the return stays finite.

An everyday picture

Think of a promise of money. A dollar you hold today is worth more than a promised dollar next year, because you could invest the one you hold, and because promises can fall through. The discount factor is how much less you trust a distant reward compared with a near one. Each step of waiting multiplies its value by the same fraction.

Worked example: A robot's rewards over four steps

A cleaning robot earns points after each of its next four steps: 1, 0, 2 and 5 points, and nothing after that. We use the discount factor $\gamma = 0.9$ and start counting at time $t = 0$ , so these are the rewards $r_1$ to $r_4$ .

Name each reward and its distance. The first reward, 1 point, is the very first in the sum, so $k = 0$ . The second reward, 0 points, has $k = 1$ . The third, 2 points, has $k = 2$ , and the fourth, 5 points, has $k = 3$ . Each $k$ tells us how many steps after the first reward that reward arrives.
Discount each reward. Multiply each reward by $0.9^{k}$ : $1 \times 0.9^{0} = 1$ , $0 \times 0.9^{1} = 0$ , $2 \times 0.9^{2} = 1.62$ and $5 \times 0.9^{3} = 3.645$ . The 5-point reward is three steps after the first one, so its weight is $0.729$ , and it counts for 3.645 points.
Add the four values. The return is $G_0 = 1 + 0 + 1.62 + 3.645 = 6.265$ points. Each term is already a reward multiplied by its weight, so the sum is the total value of the robot’s future from time 0.
Check with the shortcut, starting from the end. Start from the last reward: $G_3 = 5$ . Then $G_2 = 2 + 0.9 \times 5 = 6.5$ , $G_1 = 0 + 0.9 \times 6.5 = 5.85$ , and finally $G_0 = 1 + 0.9 \times 5.85 = 6.265$ . Each step adds one reward and discounts the later total once, so the shortcut matches the long sum.
Read the result. The discounted return is 6.265 points. The 5-point reward is worth 3.645 points in today’s terms, so the discount cut its value by about 27 percent.

Answer: The discounted return is 6.265 points. The 5-point reward arrives three steps after the first one, so it is multiplied by $0.9^3 = 0.729$ and counts for 3.645 points. The shortcut gives the same total.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Starting the sum at $r_t$ instead of $r_{t+1}$ . The reward for the step taken at time $t$ arrives after that step, so the sum starts one index later.
Using $\gamma = 1$ for a task that never ends. The sum can then grow without limit when rewards keep coming.
Starting the discount at $\gamma^1$ . The first reward in the sum is multiplied by $\gamma^0 = 1$ , so it is not discounted at all.
Comparing returns computed with different discount factors. Changing $\gamma$ changes the numbers, so the values are not on the same scale.

Where you will meet it

The target in temporal-difference learning, where the return is estimated from one reward plus the estimated value of the next state.
Policy-gradient methods, which teach the agent’s action rule by giving more credit to the actions that were followed by a high return.
Game-playing and robot agents that must trade off a reward now against rewards later.
Business valuation, where a stream of future payments is discounted to a value today, using the same kind of sum.

Questions people ask

What does a discount factor of 0 mean?

The agent cares only about the next reward. Every later term is multiplied by $0^{k} = 0$ for $k \geq 1$ , so the return is $r_{t+1}$ alone.

Why must $\gamma$ stay below 1?