TF-IDF

Q: What if a word is not in the collection at all?

Then $\text{df}(t) = 0$, and the ratio divides by zero. Real systems avoid this by scoring only terms that appear in the collection, or by adding 1 to the document frequency.

In one sentence: TF-IDF gives a word a high score in a document when the word appears often there but rarely across all the other documents.

$\text{tfidf}(t, d) = \text{tf}(t, d) \times \ln\frac{N}{\text{df}(t)}$

Read it out loud: “tf-idf of t and d equals t f of t and d, times the natural log of N divided by d f of t. The score is written tfidf of t, d. t is the term, the word being scored. d is one document. t f of t, d is the term frequency, how many times the word appears in that document. N is the number of documents in the whole collection. d f of t is the document frequency, how many documents contain the word at least once. ln is the natural logarithm, a function described below.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Term, document and collection: A term is the word or phrase being scored, such as the word saffron. A document is one piece of text, such as one recipe. A collection is the whole set of documents being compared, such as 100 recipes. The formula scores one term in one document, but it looks at the whole collection to decide how special that term is.
Term frequency and document frequency: These are two different counts. The term frequency counts how many times a word appears in one document, so a word that appears 3 times has a term frequency of 3. The document frequency counts how many documents contain the word at least once across the collection, no matter how many times it appears in each one.
Ratios and the division $N \div \text{df}$: A ratio compares two numbers by division. The ratio $\frac{N}{\text{df}(t)}$ divides the number of documents by the number of documents that contain the term. If there are $N = 100$ recipes and saffron appears in 4 of them, the ratio is $100 \div 4 = 25$ . The ratio is large for a rare word and equals 1 for a word found in every document.
The natural logarithm $\ln$ and the number $e$: The number $e$ is a fixed number, about 2.71828, that appears wherever something grows or shrinks in proportion to its size. The natural logarithm, written $\ln$ , asks which power of $e$ gives a number. So $\ln 25 \approx 3.2189$ because $e$ raised to about 3.2189 is 25. The logarithm shrinks very large ratios, so a very rare word does not swamp the score.
Products, like $\text{tf} \times \ln$: A product is the result of multiplying. Two quantities written side by side, or joined by $\times$ , are multiplied together. The TF-IDF score is the term frequency times the logarithm, so it is high only when both parts are high. A zero in either part gives a zero score.

What each symbol means

Symbol	Meaning
$t$	The term: the word or phrase being scored, such as the word saffron. It is a piece of text, not a number, so it has no unit. Each term is scored separately in each document.
$d$	One document, such as one recipe or one article. It is a piece of text, so it has no unit. Each document gets its own TF-IDF score for every term it contains.
$N$	The total number of documents in the collection, a whole number with no unit. The example uses 100 recipes, so $N = 100$ .
$\text{tf}(t, d)$	The term frequency: how many times the term $t$ appears in document $d$ . It is a plain count with no unit. The example recipe uses saffron 3 times, so the term frequency of saffron in that recipe is 3.
$\text{df}(t)$	The document frequency: how many documents in the whole collection contain the term $t$ at least once. It is a plain count with no unit. In the example, saffron appears in 4 of the 100 recipes, so $\text{df} = 4$ .
$\text{tfidf}(t, d)$	The TF-IDF score of term $t$ in document $d$ . It is a number with no unit, and a higher score means the word says more about that document than about the others. The example gives saffron a score of about 9.66 in its recipe.
$\ln$	The natural logarithm, a function that asks which power of $e$ , about 2.718, gives the number inside the brackets. It has no unit. The example uses $\ln 25 \approx 3.2189$ , because $e$ raised to about 3.2189 is 25.

The formula piece by piece

$\text{tf}(t, d)$: The first part: the term frequency. It counts how many times the word appears in this one document. A word that appears more often in a document is more about that document, so this part grows with the count. In the example it is 3 for saffron.
$\frac{N}{\text{df}(t)}$: The rarity ratio: the number of documents divided by the number of documents that contain the term. The ratio is large when few documents have the word. For saffron it is $\frac{100}{4} = 25$ , and for the word the it is $\frac{100}{99} \approx 1.0101$ , because almost every recipe contains it.
$\ln\frac{N}{\text{df}(t)}$: The logarithm of that ratio. This part is called the inverse document frequency, or idf. A word that appears in every document has a ratio of 1, and $\ln 1 = 0$ , so it adds nothing. For saffron, $\ln 25 \approx 3.2189$ .
$\text{tf}(t, d) \times \ln\frac{N}{\text{df}(t)}$: The right-hand side of the formula: the term frequency multiplied by the idf. The product rewards a word that is frequent in this document and rare across the collection. In the example, $3 \times 3.2189 \approx 9.66$ for saffron.
$\text{tfidf}(t, d) = \text{tf}(t, d) \times \ln\frac{N}{\text{df}(t)}$: The whole formula as one sentence: the TF-IDF score of a term in a document equals how many times the term appears in that document, times the natural logarithm of the number of documents divided by the number of documents that contain the term.

What the formula is doing, and why

Search engines need a way to tell which words matter in a document. Common words such as the and and appear in almost every document, so they say little about any one of them. A word such as saffron that appears often in one recipe and rarely in the others says a lot about that recipe. TF-IDF combines these two ideas into one score.

The first part, term frequency. $\text{tf}(t, d)$ counts how many times the term appears in the document. More mentions mean the term is more about that document. Other versions divide by the length of the document or use a different scale, but this page uses the plain count.

The second part, rarity. The inverse document frequency is $\ln\frac{N}{\text{df}(t)}$ . The ratio $\frac{N}{\text{df}(t)}$ is large when few documents contain the term, so the score grows as a word gets rarer. A word that appears in every document has the ratio $\frac{N}{N} = 1$ , and $\ln 1 = 0$ , so it counts for nothing. The logarithm keeps a very rare word from swamping everything else.

Why multiply? A word must be frequent in this document and rare across the collection to score high. Multiplying the two parts means a zero in either part gives a zero score. A word that appears nowhere in the document scores nothing, and so does a word that appears in every document. The natural logarithm here has base $e \approx 2.718$ . Some tools use base 10 or add smoothing, which changes the numbers but not the idea.

When does it apply? It applies to a collection of documents that you can count. The numbers depend on the collection, so the same word can score differently in another set of documents. Compare scores only inside one collection.

An everyday picture

Think of a class in which every student writes an essay on a topic. The word photosynthesis, used ten times in one biology essay, is a strong clue to that essay’s subject, because most other essays do not use it. The word the appears in every essay and tells you nothing about the subject. TF-IDF rewards the first word and gives the second a score close to zero.

Worked example: The word saffron in 100 recipes

A collection has $N = 100$ recipes. The word saffron appears in $\text{df} = 4$ of them, and the word the appears in 99 of them. In one recipe $d$ , saffron appears 3 times, so $\text{tf} = 3$ , and the word the appears 5 times. The collection is made up for this example. The logarithm is the natural log, $\ln$ , with base $e$ .

Write down the counts. For saffron, $\text{tf} = 3$ in recipe $d$ and $\text{df} = 4$ across the 100 recipes. For the word the, $\text{tf} = 5$ in recipe $d$ and $\text{df} = 99$ across the collection. Both words are scored in the same recipe, so the two scores can be compared fairly.
Find the ratio for saffron. $\frac{N}{\text{df}} = \frac{100}{4} = 25$ . That means 25 recipes for every recipe that uses saffron, or saffron appears in only 1 recipe out of every 25. A large ratio means a rare word.
Take the natural log. $\ln 25 \approx 3.2189$ . The log asks which power of $e$ , about 2.718, gives 25. Because $e$ raised to about 3.2189 is 25, the log of 25 is about 3.2189. The log turns the large ratio into a smaller number that is easier to combine with the count.
Multiply by the term frequency for saffron. $\text{tfidf} = 3 \times 3.2189 = 9.6566 \approx 9.66$ . Saffron is used 3 times in this recipe and is rare across the collection, so it gets a high score.
Find the score for the. The ratio is $\frac{100}{99} \approx 1.0101$ . Its natural log is $\ln 1.0101 \approx 0.01005$ , which is very close to 0, because 99 of the 100 recipes already contain the word. Multiplying by its count of 5 gives $5 \times 0.01005 \approx 0.0503$ .
Compare the two scores. Saffron scores about 9.66, and the scores about 0.0503. Saffron is about 192 times as high, because it is rare across the collection while the is in almost every recipe. The ratio of the two scores is $9.66 \div 0.0503 \approx 192$ .
Read the result plainly. The word saffron is a clue to what this recipe is about, while the word the tells you almost nothing. TF-IDF gives saffron a high score in this recipe for exactly that reason, and it gives the a score near zero.

Answer: The word saffron scores about 9.66 in this recipe, and the word the scores about 0.0503. Saffron scores high because it is frequent in this recipe and rare across the collection. The word the scores almost zero, because it appears in nearly every recipe.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Mixing up the two counts. Term frequency counts how many times a word appears in one document, and document frequency counts how many documents contain it across the whole collection.
Flipping the ratio. It must be $N$ divided by $\text{df}$ , so that rare words get a large number. Dividing $\text{df}$ by $N$ would make rare words score near zero.
Expecting every tool to give the same numbers. Some versions use other logs, smoothing or normalized term frequencies, so check which version is in use before comparing scores.
Thinking a word in every document is important. Its ratio is 1 and its logarithm is $\ln 1 = 0$ . That is correct, because such a word cannot tell documents apart.

Where you will meet it

Ranking documents for a search query in classic search systems, where matching words are weighted by their TF-IDF scores.
Finding keywords in a document by picking the terms with the highest scores.
Text classification and document clustering, where each document becomes a list of TF-IDF numbers, one per word, which a model uses to sort documents into groups.
Comparing how similar two documents are, by comparing their lists of TF-IDF numbers with a measure called cosine similarity, which checks whether two lists point in the same direction.

Questions people ask

Why is there a logarithm?

Without it, the ratio $\frac{N}{\text{df}}$ can become very large, so a rare word would completely dominate the score. The logarithm shrinks the ratio, so the rarity part stays on a scale close to the term frequency. For example, the ratio 25 becomes about 3.2 after the log.

What if a word is not in the collection at all?

Then $\text{df}(t) = 0$ , and the ratio divides by zero. Real systems avoid this by scoring only terms that appear in the collection, or by adding 1 to the document frequency.

Is TF-IDF the same as a word count?

No. A word count uses only the term frequency. TF-IDF also lowers the score of words that appear in many documents, so common words count for less than rare, meaningful ones.

PageRank: PageRank gives each web page a score based on which other pages link to it, and a link from a page with a high score counts for more.
Shannon Entropy: Shannon entropy is one number, measured in bits, that says how unpredictable a source of messages is on average: the more predictable it is, the fewer bits its messages need.