Linear Regression

In one sentence: Linear regression predicts a number by drawing the straight line that sits closest to the data points, so you can read off a guess for any new input.

$\hat{y} = w x + b$

Read it out loud: “y hat equals w times x, plus b. The predicted y is the slope w times the input x, plus the intercept b. The hat over y means the value is a prediction, not a measurement.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Slope and intercept of a straight line: A straight line can be written as $y = w x + b$ . The slope $w$ says how much $y$ rises each time $x$ goes up by one. A slope of 0.7 means a step of 1 in $x$ adds 0.7 to $y$ . The intercept $b$ is the height of the line where $x = 0$ , the point where the line crosses the vertical axis. A slope of 0 is a flat line, and a negative slope is a line that goes downhill.
A bar over a letter, like $\bar{x}$ , means average: A bar placed over a letter, read ‘x bar’, is the average of that list. So $\bar{x}$ is the average of all the $x$ values. To find it, add up the values and divide by how many there are. The average of 1, 2, 3 and 4 is $\frac{1 + 2 + 3 + 4}{4} = 2.5$ . The average is the balance point of the list, the number that the values spread around.
The sum sign $\sum$ over data points: The Greek capital sigma, $\sum$ , means add up the list. $\sum_{i}$ means add up over every data point, where $i$ counts the points: point 1, point 2, and so on. The sum sign is shorthand for writing every term out in full and adding them, so each formula stays short even when there are many data points.
Deviations from the average, like $x_i - \bar{x}$: A deviation is how far a value sits from the average. If the average hours studied is 2.5 and a student studied 1 hour, the deviation is $1 - 2.5 = -1.5$ , a negative number because the student is below average. Deviations are positive above the average, negative below it, and they always add up to 0 across the whole list.
Residuals and least squares: A residual is the miss for one data point: the observed value minus the predicted value, $y_i - \hat{y}_i$ . A line that fits well has small residuals. Least squares is the rule for picking the best line. It squares each residual, adds the squares, and chooses the slope and intercept that make the total as small as possible. Squaring keeps positive and negative misses from canceling out.

What each symbol means

Symbol	Meaning
$\hat{y}$	The predicted output, the value the line gives for a chosen input $x$ . The hat marks it as a prediction rather than a measured value. It has the same unit as $y$ , such as points out of 10 for a quiz score.
$x$	The input, a number you already know, such as hours of study. It has whatever unit the input is measured in, for example hours. The line uses $x$ to produce the prediction, so every guess starts from a chosen value of $x$ .
$w$	The slope, also called the weight. It says how much the prediction changes when $x$ goes up by one unit. Its unit is the unit of $\hat{y}$ divided by the unit of $x$ , so a slope of 0.7 points per hour means each extra hour adds 0.7 points.
$b$	The intercept, also called the bias. It is the prediction when $x$ is 0, and it sets the height where the line crosses the vertical axis. Its unit is the same as the unit of $\hat{y}$ , such as points on the quiz score scale.

The formula piece by piece

$w x$: Start with the input scaled by the slope. Multiplying $x$ by $w$ says how far the prediction moves for this input, because every extra unit of $x$ adds another $w$ to the result. A bigger $x$ gives a bigger change, at a steady rate.
$w x + b$: Add the intercept to the scaled input. The intercept lifts or drops the whole line, so the prediction starts at the height $b$ before the input changes it. Together these two parts give the height of the straight line above any input.
$\hat{y} = w x + b$: The whole formula in one sentence: the predicted output equals the slope times the input, plus the intercept, so every input gives one point on a straight line.
$w = \frac{\sum_{i} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i} (x_i - \bar{x})^2}$: The standard least-squares formula for the best slope, written with the averages. The top adds up how the inputs and outputs move together, and the bottom adds up how much the inputs spread out. Dividing the first by the second gives the slope.

What the formula is doing, and why

Linear regression is the simplest way to use one number to predict another. You start with pairs of values: an input $x$ and the output $y$ that you actually observed. The formula $\hat{y} = w x + b$ describes a straight line. For any input $x$ , multiply by the slope $w$ , add the intercept $b$ , and you get the predicted output $\hat{y}$ .

Why choose the least-squares line? The hard part is choosing $w$ and $b$ . For each data point, the residual is the observed value minus the predicted value, $y_i - \hat{y}_i$ . Least squares picks the line that makes the sum of the squared residuals as small as possible. Squaring stops positive and negative misses from canceling out, and it punishes big misses more than small ones.

Why the slope formula looks like that? Let $\bar{x}$ and $\bar{y}$ be the averages of the inputs and the outputs. The top of the slope formula adds up, over every point, the product of the two deviations, $(x_i - \bar{x})(y_i - \bar{y})$ . Points that are above average in both values push the sum up, and points that are below average in both also push it up. The bottom adds up the squared input deviations, which is how much the inputs spread out. The intercept then follows from the averages: $b = \bar{y} - w \bar{x}$ . The fitted line always passes through the point $(\bar{x}, \bar{y})$ .

What does the slope tell you? A positive slope means the output tends to rise as the input rises. A negative slope means it tends to fall. A slope near zero says the input tells you little about the output. The line is only as good as the data behind it: when the real relationship curves, a straight line will miss the pattern.

An everyday picture

Think of a teacher with a scatter of dots on graph paper, one dot per student, with hours studied across the bottom and quiz score up the side. Linear regression is the ruler laid across the dots so the line sits as close to all of them as possible. To estimate a new student’s score, find their hours along the bottom and read the height of the ruler above that spot.

Worked example: Predicting a quiz score from hours of study

Four students report their study hours $x$ and get a quiz score $y$ out of 10. The data are $x = (1, 2, 3, 4)$ hours and $y = (2, 4, 5, 4)$ points. We will find the best line and then predict the score for a student who studies 5 hours. The four points are far too few for a real study, so treat this as a demonstration of the method.

Find the average hours. The average of the inputs is $\bar{x} = \frac{1 + 2 + 3 + 4}{4} = 2.5$ hours. This is the balance point of the study times, and it is the first thing the slope formula needs.
Find the average score. The average of the outputs is $\bar{y} = \frac{2 + 4 + 5 + 4}{4} = 3.75$ points. The line will always pass through the point made of both averages, $(2.5, 3.75)$ .
Find each deviation from the average. For the hours, $1 - 2.5 = -1.5$ , $2 - 2.5 = -0.5$ , $3 - 2.5 = 0.5$ and $4 - 2.5 = 1.5$ . For the scores, $2 - 3.75 = -1.75$ , $4 - 3.75 = 0.25$ , $5 - 3.75 = 1.25$ and $4 - 3.75 = 0.25$ . Negative deviations mean below average.
Add up the products of the deviations. The top of the slope formula is $\sum (x_i - \bar{x})(y_i - \bar{y})$ . Multiplying the matching deviations gives $(-1.5)(-1.75) = 2.625$ , $(-0.5)(0.25) = -0.125$ , $(0.5)(1.25) = 0.625$ and $(1.5)(0.25) = 0.375$ . Their total is $3.5$ , which says the hours and scores move together.
Add up the squared hour deviations. The bottom of the slope formula is $\sum (x_i - \bar{x})^2$ , which measures how much the hours spread out. We get $2.25 + 0.25 + 0.25 + 2.25 = 5.0$ .
Divide to get the slope. The slope is $w = \frac{3.5}{5.0} = 0.7$ points per hour. Each extra hour of study is linked, on average, with about 0.7 more points on the quiz.
Find the intercept. The intercept is $b = \bar{y} - w \bar{x} = 3.75 - 0.7 \times 2.5 = 3.75 - 1.75 = 2.0$ points. This is the predicted score for zero hours of study, and it places the line so that it passes through the averages.
Predict for 5 hours. Put $x = 5$ into the line: $\hat{y} = 0.7 \times 5 + 2.0 = 3.5 + 2.0 = 5.5$ points. The prediction is made by reading the height of the line above 5 hours on the bottom axis.

Answer: The best line is $\hat{y} = 0.7x + 2.0$ . A student who studies 5 hours is predicted to score about 5.5 points out of 10. Each extra hour adds about 0.7 points to the prediction. Four data points is far too few for a real study, so treat this as a demonstration of the method.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Swapping the input and the output. The line predicts $y$ from $x$ , so the value you know goes in as $x$ and the value you want to predict is $\hat{y}$ .
Forgetting to subtract the averages in the slope formula. Using the raw $x_i$ and $y_i$ gives a different, wrong slope.
Predicting far outside the data. A line fitted to 1 to 4 hours of study says little about 40 hours, because the pattern may not continue that far.
Expecting a straight line to fit curved data. Linear regression always draws a straight line, even when the real pattern bends.

Where you will meet it

Predicting house prices from floor area, where the slope gives a rough price per square meter.
Forecasting sales, demand or energy use from a steady trend over time.
Calibrating sensors, such as turning a voltage reading into a temperature by fitting a line to known reference points.
Serving as the baseline model in statistics and data science courses, which more complex models must beat.

Questions people ask

What does least squares mean?

It is the rule for choosing the line. Among all straight lines, pick the one that makes the sum of the squared vertical distances from the points to the line as small as possible.

Is linear regression the same as correlation?

No. Correlation measures how tightly the points cluster around a line, on a scale from $-1$ to $1$ . Regression gives the actual line, with a slope and an intercept you can use to predict.

How is linear regression different from logistic regression?

Linear regression predicts a number such as a score or a price. Logistic regression predicts the probability of a yes-or-no outcome, using an S-shaped curve.

Why is it called linear?

The prediction changes at a constant rate. Each extra unit of $x$ adds the same amount, $w$ , to the prediction, which is exactly what a straight line does.

Mean Squared Error (MSE): Mean squared error measures how wrong a model’s guesses are by squaring each miss, so big misses count much more, and then averaging all of the squared misses.
Gradient Descent: Gradient descent nudges a model’s settings a small step in the direction that makes the error smaller, then repeats that step again and again until the error stops falling.
Logistic Regression: Logistic regression takes the same weighted sum as a straight line, then squeezes it through an S-shaped curve so the result is a probability between 0 and 1 for a yes-or-no question.
L2 Regularization: L2 regularization adds a penalty for large weights, so the model prefers small, moderate weights that tend to hold up on new data.