Average Memory Access Time

In one sentence: AMAT is the average time it takes a computer to fetch one piece of data, when most fetches are quick and a few are slow.

$\text{AMAT} = t_{\text{hit}} + r_{\text{miss}} \times t_{\text{penalty}}$

Read it out loud: “AMAT equals t sub hit plus r sub miss times t sub penalty. The small words after a letter are subscripts, labels that say which quantity it is. t sub hit is the hit time, the time for a quick read. r sub miss is the miss rate, the fraction of reads that are slow. t sub penalty is the miss penalty, the extra time one slow read costs. The plus sign adds the hit time to the product, and the times sign multiplies the miss rate by the penalty. All the times must use the same unit, such as nanoseconds.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Powers of ten and the nanosecond (ns): Computer speeds are measured in tiny fractions of a second. A nanosecond (ns) is one billionth of a second, which is $10^{-9}$ s. The small raised number $-9$ is an exponent, and it means divide by ten nine times: $10^{-9} = 0.000000001$ , a decimal point, eight zeros and then a 1. So 6 ns is $6 \times 10^{-9}$ s, six billionths of a second.
Cache, hit and miss: A cache is a small, fast store placed close to the processor, the chip that runs the program. When the processor needs a piece of data, it checks the cache first. If the data is there, that is a hit. If it is not there, that is a miss, and the data must be fetched from the slower main memory, which takes longer.
Percent as a fraction, like 5% = 0.05: A percent means ‘out of one hundred’. To turn a percent into a decimal fraction, divide by 100, which moves the decimal point two places to the left. So 5% becomes 0.05, and 2.5% becomes 0.025. The formula needs the fraction, because it multiplies the miss rate by a time, and using 5 instead of 0.05 would make the answer 100 times too big.
Average: An average is a total divided by the number of items. If five reads take 1, 1, 1, 1 and 11 ns, the total is $15$ ns and the average is $15 \div 5 = 3$ ns. AMAT is an average over many reads, so one slow read does not make the average slow, but a steady share of slow reads adds up.
Subscripts, like $t_{\text{hit}}$: A small label written below a letter is a subscript. In $t_{\text{hit}}$ the letter $t$ stands for time, and the subscript names which time it is: the hit time. Read $t_{\text{hit}}$ as ‘t-hit’. The subscript is a name, not a multiplication, so $t_{\text{hit}}$ is one single quantity.

What each symbol means

Symbol	Meaning
$\text{AMAT}$	The average memory access time: the average time one memory read takes, counting both hits and misses across many reads. It is measured in time units such as nanoseconds (ns), where 1 ns is a billionth of a second. The example below gives 6 ns, which is six billionths of a second.
$t_{\text{hit}}$	The hit time: how long it takes to read data that is already in the cache. It is measured in nanoseconds (ns) or in clock cycles. Every read pays it, because the cache is always checked first. The example uses 1 ns.
$r_{\text{miss}}$	The miss rate: the fraction of reads that are not found in the cache. It is a number from 0 to 1 with no unit, so a miss rate of 0.05 means 5 misses in every 100 reads. In percent form, 0.05 is the same as 5%.
$t_{\text{penalty}}$	The miss penalty: the extra time a miss costs, from the moment the cache misses until the data arrives from the slower main memory. It is measured in nanoseconds (ns), like the hit time. The example uses 100 ns for each fetch from main memory.
$\times$	The multiplication sign. It multiplies the miss rate, a fraction with no unit, by the miss penalty in nanoseconds. Only misses pay the penalty, so this product is the average extra time that every read carries.

The formula piece by piece

$t_{\text{hit}}$: Start with the cost every read pays. The hit time is the time to check the cache and read the data it holds. Every read pays it, whether it is a hit or a miss, so it appears once in the formula. In the example it is 1 ns, one billionth of a second.
$r_{\text{miss}}$: The miss rate is the share of reads that miss. It is a fraction from 0 to 1 with no unit, so 0.05 means 5 misses in every 100 reads. Multiplying it by a time turns the share into an average time, which is the next step.
$r_{\text{miss}} \times t_{\text{penalty}}$: The average extra time per read. Only the misses pay the penalty, so multiplying the share of misses by the penalty spreads that cost across all reads. In the example it is $0.05 \times 100 = 5$ ns for every read, on average.
$\text{AMAT} = t_{\text{hit}} + r_{\text{miss}} \times t_{\text{penalty}}$: The whole formula as one sentence: the average memory access time equals the hit time, plus the miss rate times the miss penalty. Every read pays the hit time, and the misses add their share of the penalty on top.

What the formula is doing, and why

A processor reads data through a cache, a small and fast store placed close to the processor. Each read is either a hit or a miss. A hit finds the data in the cache and takes $t_{\text{hit}}$ . A miss does not find it, so the read must wait for the slower main memory. A miss therefore costs the hit time plus the extra miss penalty, $t_{\text{penalty}}$ . Every read starts by checking the cache, so every read pays $t_{\text{hit}}$ .

Why multiply? Only some reads miss, so only some reads pay the penalty. Multiplying the miss rate by the penalty gives the average extra time per read. Suppose 5 reads out of every 100 miss, and each miss costs an extra 100 ns. The extra time over those 100 reads is $5 \times 100 = 500$ ns. Spreading 500 ns over 100 reads gives an average of $500 \div 100 = 5$ ns per read. That is exactly $r_{\text{miss}} \times t_{\text{penalty}}$ .

Why add the hit time? Every read pays the hit time, because the cache must be checked before anything else happens. So the average is the hit time plus the average extra time from the misses. Written together, $\text{AMAT} = t_{\text{hit}} + r_{\text{miss}} \times t_{\text{penalty}}$ . Misses are rare but expensive, so the miss rate often matters most. In the example below, a 5% miss rate adds 5 ns, which is five times the 1 ns hit time. Cutting the miss rate in half, to 2.5%, would bring the average from 6 ns down to 3.5 ns.

When does it apply? The formula describes one cache in front of one slower memory. It is an average over many reads, not the time of any single read. Each read takes either the hit time, or the hit time plus the penalty. Real computers often have several cache levels, each checked after the one before it misses, and the same reasoning is applied to each level.

An everyday picture

Think of a cook with a spice rack on the counter and a storeroom down the hall. Taking a spice from the counter takes a few seconds, which is the hit time. Sometimes the spice is not on the counter, so the cook walks to the storeroom, and that extra walk is the miss penalty. The average time to get one spice is the counter time, plus the share of trips that go to the storeroom, times the length of the walk. The counter is the cache, the storeroom is main memory, and the share of trips is the miss rate.

Worked example: A cache that misses 5% of the time

Imagine a laptop program that reads memory 1,000 times. The numbers are made up for this example, to show how the formula works. They are not measurements of a real chip. The cache answers in $t_{\text{hit}} = 1$ ns. Five percent of reads miss, so $r_{\text{miss}} = 0.05$ . That means 50 misses and 950 hits in 1,000 reads. A miss costs an extra $t_{\text{penalty}} = 100$ ns to fetch the data from main memory.

Write the formula and put in the numbers. The formula says the average time equals the hit time plus the miss rate times the miss penalty. We replace each name with its value: $\text{AMAT} = 1 + 0.05 \times 100$ ns. Every time uses nanoseconds, so the answer will also be in nanoseconds.
Turn the percentage into a fraction. The miss rate is 5%, and 5% means 5 out of every 100 reads. Dividing by 100 gives $0.05$ . We need the fraction because the formula multiplies the rate by a time. Multiplying by 5 instead would make the penalty term 100 times too large.
Find the average extra time from misses. Multiplying the rate by the penalty gives $0.05 \times 100 = 5$ ns. Here is the same idea counted in whole reads. Out of 1,000 reads, 50 miss, and each miss pays 100 ns extra, a total of $50 \times 100 = 5000$ ns. Spreading 5000 ns over 1000 reads gives $5000 \div 1000 = 5$ ns per read.
Add the hit time. Every read pays the 1 ns hit time, so we add it to the 5 ns of average extra time: $\text{AMAT} = 1 + 5 = 6$ ns. The 1 ns is paid by every read, which is why it is added once and not multiplied by the miss rate.
Check the answer with all 1,000 reads. There are 950 hits, and each takes 1 ns. There are 50 misses, and each takes the hit time plus the penalty, which is $1 + 100 = 101$ ns. The total is $950 \times 1 + 50 \times 101 = 950 + 5050 = 6000$ ns. Dividing the total by 1000 reads gives $6000 \div 1000 = 6$ ns, the same answer as the formula.
Read the size of the time. One nanosecond is $10^{-9}$ s, so 6 ns is $6 \times 10^{-9}$ s. Light travels about $3 \times 10^{8}$ meters every second. Adding exponents when multiplying powers of ten, $10^{8} \times 10^{-9} = 10^{-1}$ , so the light covers $3 \times 6 \times 10^{-1} = 1.8$ meters in that time. A 6 ns read is therefore a very short interval, in which light covers about the height of a tall adult.
Test a better cache. If the miss rate fell to 2.5%, which is $0.025$ , the formula gives $1 + 0.025 \times 100 = 1 + 2.5 = 3.5$ ns. Halving the misses cut the average from 6 ns to 3.5 ns, a saving of 2.5 ns on every read.

Answer: The average memory read takes 6 ns, which is six billionths of a second. That is six times the 1 ns hit time. Only 5% of reads miss, but those misses add 5 ns to the average. A miss rate of 2.5% would bring the average down to 3.5 ns.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Writing the miss rate as a whole number. Use $0.05$ for 5%, not 5. With 5 instead, the penalty term becomes $5 \times 100 = 500$ ns instead of $0.05 \times 100 = 5$ ns, which is 100 times too large.
Leaving out the hit time. The formula adds $t_{\text{hit}}$ to every read, including the reads that hit, because the cache is always checked first.
Using the hit rate where the miss rate belongs. A 95% hit rate means a 5% miss rate, so the formula needs $r_{\text{miss}} = 0.05$ , not 0.95.
Mixing units, such as nanoseconds for the hit time and clock cycles for the penalty. Convert every time to one unit before you multiply and add.

Where you will meet it

Comparing cache sizes and designs in computer architecture, to see whether a bigger cache lowers the average time enough to be worth its cost.
Designing processors with several cache levels, where each level is checked only after the level before it misses.
Tuning software for memory access, since programs that reuse nearby data find it in the cache more often, which lowers the miss rate.
Explaining why a program slows down when its data stops fitting in the cache, because the miss rate rises and the average time grows with it.

Questions people ask

What is a cache miss penalty?

It is the extra time spent waiting for data that is not in the cache. It is counted from the moment the cache misses until the data arrives from the next, slower level of memory, such as main memory. In the example it is 100 ns, which is 100 billionths of a second.

Why does the miss rate multiply the penalty?

Only misses pay the penalty, so the average penalty is the fraction of reads that miss, times the penalty. If 5 reads out of every 100 each pay 100 ns, the total extra time is $5 \times 100 = 500$ ns spread over 100 reads. That is an average of $500 \div 100 = 5$ ns per read.

Does AMAT tell me how long each read takes?

No. It is an average over many reads. Each single read takes either the hit time, which is 1 ns in the example, or the hit time plus the penalty, which is 101 ns. The average of 6 ns mixes these two kinds of read.

CPU Time Equation: The CPU time equation says how many seconds a processor spends on a program, using three counts: the steps the program takes, the clock ticks each step needs, and how fast the clock ticks.
Amdahl's Law: Amdahl’s law tells you how much faster a whole program gets when you speed up only part of its work, and why the part you leave alone caps the gain.