Z-Score Standardization

In one sentence: A z-score tells how far a value is from the average, counted in typical steps of spread, so features measured in different units can be compared fairly.

$z = \frac{x - \mu}{\sigma}$

Read it out loud: “z equals x minus mu, all divided by sigma. The letter z is the z-score, the new value after standardizing. The letter x is the original value. The Greek letter mu, said MYOO, is the mean, the average. The Greek letter sigma, said SIG-muh, is the standard deviation, the typical distance of the values from the mean. In words: the z-score is the distance from the mean, divided by the standard deviation.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

The mean, $\mu$: The mean, or average, is all the values added together and divided by how many there are. The Greek letter $\mu$ , called mu and said MYOO, is the usual name for the mean of a feature. For the hours 7, 9, 10, 11 and 13, the mean is $\frac{7 + 9 + 10 + 11 + 13}{5} = 10$ hours.
Standard deviation, $\sigma$: The standard deviation, written $\sigma$ and said SIG-muh, is the typical distance of the values from the mean. To find it, measure each value’s distance from the mean, square each distance, average the squares, and then take the square root of that average. The square root undoes squaring: $\sqrt{4} = 2$ , because $2 \times 2 = 4$ .
Deviation, a distance with a sign: A deviation is a value minus the mean. It keeps its sign. If a value of 13 has a mean of 10, its deviation is $13 - 10 = +3$ , which means 3 above the average. A value below the mean has a negative deviation. Squaring removes the sign, which is why the standard deviation is built from squares.
Square roots: The square root, written $\sqrt{x}$ , asks which number multiplied by itself gives $x$ . So $\sqrt{9} = 3$ , because $3 \times 3 = 9$ . Taking the square root undoes the squaring done earlier, so the standard deviation returns to the same units as the data, such as hours.
Feature and scaling: A feature is one measured property of each example that a model uses as an input, such as hours studied or a salary in dollars. Scaling a feature means changing its numbers so that features with different units become comparable. Standardizing is one way to scale, and it is also called a z-score transform.
Training data and test data: A model learns from training data, a set of examples used to find its settings. Test data is a separate set that is held back, to check how the model does on examples it has never seen. Keeping the two apart matters for the mean and the standard deviation, as the explanation below shows.

What each symbol means

Symbol	Meaning
$z$	The z-score, the new value after standardizing. It is a plain number with no unit: how many standard deviations $x$ sits above or below the mean. Positive means above average, and negative means below.
$x$	The original value of one example for this feature, in its own units, such as hours or dollars. For a student who studies 13 hours, $x = 13$ hours.
$\mu$	The mean of the feature, the average over the training data. It is in the same units as $x$ , such as hours, and it is the center that the z-score is measured from.
$\sigma$	The standard deviation, the typical distance of the values from the mean. It is in the same units as $x$ , such as hours, and it must be greater than zero, because dividing by zero is not allowed.

The formula piece by piece

$x$: The original value of one example, in its own units. For a student who studies 13 hours a week, $x = 13$ hours.
$x - \mu$: The deviation: the value minus the mean. It is the signed distance from the average, in hours. Here $13 - 10 = 3$ hours above the average.
$\sigma$: The typical spread of the feature, in the same units. It is the standard deviation, which is 2 hours in this example. It measures how large a typical step away from the average is.
$\frac{x - \mu}{\sigma}$: The distance from the mean, measured in typical steps. Dividing by $\sigma$ rescales the distance, so that one unit means one typical spread. Here $\frac{3}{2} = 1.5$ typical steps.
$z = \frac{x - \mu}{\sigma}$: The z-score of a value equals its distance from the mean, divided by the standard deviation.

What the formula is doing, and why

Many learning methods work better when the input features sit on similar scales. If one feature is a salary in dollars and another is an age in years, the salary numbers are much bigger, and they dominate distances and gradient steps. A gradient step is one small adjustment a model makes while it learns. Standardizing each feature puts them on one common scale, so no feature wins because of its units alone.

Why subtract and divide? The mean $\mu$ is the average of the feature. Subtracting it moves the center of the data to zero, so values above average become positive and values below average become negative. The standard deviation $\sigma$ is the typical distance from the average. Dividing by $\sigma$ makes one typical step of spread equal to 1. So $z$ tells you how many standard deviations $x$ lies above or below the average.

Why use the training data only? The mean and the standard deviation must be computed from the training data alone. Then the same two numbers are used on the test data and on new inputs. If you compute them from the whole dataset, information from the test set leaks into training, and the results look better than they really are.

What the transform does not change. The transform changes the location and the scale of the values, but not their shape. A skewed feature stays skewed, and an outlier, an unusually extreme value, stays an outlier. Whatever the shape of the data, at most one value in nine can lie three or more standard deviations from the mean. This is Chebyshev’s inequality, a rule that holds for any shape of data.

When it applies. Standardizing helps methods that measure distances or take gradient steps, such as nearest-neighbor methods and neural networks. It does not change decision trees, which split one feature at a time. It also does not make sense for category codes such as 1 for red and 2 for blue, because those numbers have no real distance between them.

An everyday picture

Think of grading on a curve. Instead of raw scores, you report how far each student is from the class average, in units of the class’s typical spread. A student who studies 13 hours a week, in a group whose typical spread is 2 hours, sits 1.5 units above the average, whatever units the original numbers were in.

Worked example: Standardizing weekly study hours

Five students study these hours per week: $7$ , $9$ , $10$ , $11$ and $13$ . We standardize the value $x = 13$ , the hours of the last student.

Find the average. Add the five study times and divide by 5: $\mu = \frac{7 + 9 + 10 + 11 + 13}{5} = \frac{50}{5} = 10$ hours. The average is the center point that every other value is measured from.
Find each distance from the average. The deviations are $7 - 10 = -3$ , $9 - 10 = -1$ , $10 - 10 = 0$ , $11 - 10 = 1$ and $13 - 10 = 3$ hours. Squaring each one gives $9$ , $1$ , $0$ , $1$ and $9$ , which are the squared distances in squared hours.
Average the squared distances. Their total is 20, and dividing by 5 gives $\frac{20}{5} = 4$ squared hours. This average of the squares is called the variance. It is in squared hours, which is not an everyday unit, so the next step brings it back.
Take the square root. The standard deviation is $\sigma = \sqrt{4} = 2$ hours. Taking the square root undoes the squaring, so $\sigma$ is back in ordinary hours, and it is the typical distance from the average.
Standardize the value 13. Subtract the mean and divide by the standard deviation: $z = \frac{13 - 10}{2} = \frac{3}{2} = 1.5$ . The student who studies 13 hours sits 1.5 standard deviations above the group average.
Check the whole set. The five z-scores are $-1.5$ , $-0.5$ , $0$ , $0.5$ and $1.5$ . They add up to 0, so the new mean is 0. Their squares are $2.25$ , $0.25$ , $0$ , $0.25$ and $2.25$ , which average to exactly $1$ , so the new standard deviation is 1.

Answer: The z-score of 13 hours is 1.5. That student studies 1.5 standard deviations above the group average. After standardizing, the whole set has mean 0 and standard deviation 1.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Computing $\mu$ and $\sigma$ on all the data, including the test set. Fit them on the training data only.
Standardizing new data with its own mean and standard deviation. Use the numbers saved from the training data, so that every input is scaled the same way.
Reading a z-score as a probability. A z-score of 1.5 is a distance in standard deviations, not a chance of anything.
Standardizing category codes such as 1 for red and 2 for blue. Those numbers have no real distance between them, so the result means nothing.

Where you will meet it

Scaling features before distance-based methods such as k-nearest neighbors, k-means clustering and support vector machines.
Preparing inputs for gradient descent in linear, logistic and neural network models, where similar scales make training more stable.
Flagging unusual values in data pipelines, where a z-score far from zero marks a candidate outlier.
Comparing results measured on different scales, such as standard scores that put different tests on one common scale.

Questions people ask

Does z-scoring change the shape of the data?

No. It shifts and scales the values. A skewed feature stays skewed, and outliers stay far from the center.

How is this different from min-max scaling?

Min-max scaling squeezes values into a fixed range, such as 0 to 1. A z-score centers on the mean and uses standard deviations, so its range is not fixed.

Which standard deviation does scikit-learn use?

Its StandardScaler divides by the population standard deviation. That version divides by $n$ , the number of values, instead of $n - 1$ . Both versions are common, so use the same one for training and for new data.

Do I need to standardize every model's inputs?

Not always. Decision trees split one feature at a time, so scaling does not change their splits. Distance-based and gradient-based methods usually need it.

Mean Squared Error (MSE): Mean squared error measures how wrong a model’s guesses are by squaring each miss, so big misses count much more, and then averaging all of the squared misses.
R-Squared: R-squared is a score that says how much of the ups and downs in the data a model predicts correctly. A score of 1 is perfect, and 0 means the model is no better than always guessing the average.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.