Key takeaways
- Least squares minimizes squared residuals, not the number of points above or below the line.
- The slope carries units. “0.6 points per hour” is meaningful; a bare 0.6 is not.
- The intercept is often extrapolation. It is only interpretable when x = 0 is inside the observed range and makes sense.
- R² is not a quality score. A high R² can come from a wrong model, and a low R² can still hide a useful slope.
- Check residual plots before trusting inference. Linearity, constant variance, and independence matter more than the headline R².
From correlation to a prediction line
Correlation asks whether two variables move together. Regression goes further and asks how to predict one from the other. The model is a straight line, ŷ = b0 + b1x, where ŷ is the fitted value, b0 is the intercept, and b1 is the slope. The hat is important: ŷ is the line’s prediction, not the observed y.
The direction matters. Regressing y on x is not the same as regressing x on y, because each fit minimizes errors in a different direction. That asymmetry is the main practical difference from correlation, which treats the two variables symmetrically.
The least-squares formulas
The slope is the covariance of x and y divided by the variance of x:
b1 = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)²
Once the slope is known, the intercept follows from the fact that the line passes through the point (x̄, ȳ):
b0 = ȳ − b1x̄
These values minimize the residual sum of squares, SSE = Σ(y − ŷ)². Squaring is what makes the fit sensitive to large misses and gives the procedure its name. A line that is off by 2 on four points and a line that is off by 4 on one point can have the same SSE, which is why the residuals themselves deserve a look.
Worked example: the same five points
Using x = 1, 2, 3, 4, 5 and y = 2, 4, 5, 4, 5, the means are x̄ = 3 and ȳ = 4. The cross-products sum to 6 and the squared x deviations sum to 10, so b1 = 6 ÷ 10 = 0.6. The intercept is b0 = 4 − 0.6 × 3 = 2.2. The fitted line is ŷ = 2.2 + 0.6x.
Predictions are 2.8, 3.4, 4.0, 4.6, and 5.2. Residuals are therefore −0.8, +0.6, +1.0, −0.6, and −0.2, which sum to zero as least squares requires. The residual sum of squares is 0.64 + 0.36 + 1.00 + 0.36 + 0.04 = 2.40. The total sum of squares around the mean is 6.00, so R² = 1 − 2.40/6.00 = 0.60, matching the squared correlation from the same data.
Reading the slope and intercept
The slope is a rate. If x is hours studied and y is exam points, b1 = 2.5 means each additional hour is associated with 2.5 more points on average. It does not mean every student gains exactly 2.5 points, and it does not establish that studying causes the gain, because motivated students may both study more and score higher for other reasons.
The intercept is the predicted y at x = 0, which is often outside the data. A regression of weight on height might produce an intercept of −90 kg, a physically impossible value that is nonetheless the correct anchor for the line. Center x by subtracting its mean if you want an interpretable constant: the centered intercept then equals ȳ, the predicted value at average x.
What R-squared does and does not mean
R² = 1 − SSE/SST, where SST is the total variation of y around its mean. It is the proportion of variance in y explained by the fitted line, and it equals r² in simple linear regression. An R² of 0.60 means the line accounts for 60% of the sample variation; the other 40% comes from variables not in the model, measurement error, and randomness.
R² never decreases when you add a predictor, even a useless one, so it cannot be used to compare models with different numbers of predictors. Adjusted R² penalizes extra terms: R²adj = 1 − (1 − R²)(n − 1)/(n − k − 1), where k is the number of predictors. With n = 30, k = 5, and R² = 0.40, adjusted R² falls to 1 − 0.60 × 29/24 = 0.275, a much less flattering picture.
Standard errors, tests, and confidence intervals
The residual standard error is s = √(SSE/(n − 2)). For the five-point example, s = √(2.40/3) = 0.894. The standard error of the slope is:
SE(b1) = s ÷ √Σ(x − x̄)²
Here SE = 0.894 ÷ √10 = 0.283, so t = 0.6 ÷ 0.283 = 2.12 with 3 degrees of freedom. The two-sided p-value is about 0.124, and a 95% interval using t* = 3.182 runs from 0.6 − 0.90 to 0.6 + 0.90, or (−0.30, 1.50). The estimate is positive but far too imprecise to rule out no relationship, which is what five points buy you.
With a larger sample the same slope could be tightly estimated. Always report the standard error or confidence interval, because a slope without an uncertainty statement invites overconfidence.
Assumptions and diagnostics
Four assumptions support the usual tests: the relationship is linear in the parameters, the errors are independent, the errors have constant variance, and the errors are approximately normal for small-sample inference. The first three matter most for the fitted line; normality matters mainly for p-values and intervals in small samples.
Plot residuals against fitted values. A curve suggests a missing nonlinear term; a funnel suggests heteroscedasticity; a few extreme residuals suggest influential points. Plot residuals against time or observation order to check independence. A point with high leverage and a large residual can move the entire line, so investigate it rather than deleting it automatically.
Multiple regression in one paragraph
With several predictors, ŷ = b0 + b1x1 + … + bkxk. Each slope is now the change in y associated with a one-unit change in that predictor while holding the others fixed. That “holding fixed” clause is doing a lot of work: if predictors are strongly correlated with each other, their standard errors inflate and individual coefficients become unstable. Omitted variables also bias the slopes you do estimate, which is why causal language requires more than a good fit.
Common mistakes
- Interpreting the intercept when x = 0 is far outside the observed range.
- Calling R² = 0.9 “90% accurate” when it measures variance explained, not prediction error.
- Extrapolating the line beyond the data and treating the result as a forecast.
- Comparing R² across models with different numbers of predictors instead of using adjusted R².
- Ignoring a curved residual plot because the slope was statistically significant.
- Reporting a slope without units, a standard error, or a confidence interval.
Practical caveats
A regression line is a description of the observed range, not a law of nature. Predictions outside that range assume the same linear mechanism continues, which is often false. Measurement error in x biases the slope toward zero, and measurement error in y inflates the standard errors without biasing the slope.
Finally, distinguish prediction from explanation. If the goal is forecasting, judge the model on held-out error. If the goal is understanding, judge it on whether the coefficients are stable, plausible, and robust to alternative specifications. A single R² answers neither question by itself.
Frequently asked questions
What is the difference between correlation and regression?
Correlation measures the strength of a linear association and is symmetric. Regression fits a prediction line and is directional: it estimates how much y changes when x changes, in the units of the data.
Can the slope be negative?
Yes. A negative slope means y tends to fall as x rises. The sign of the slope always matches the sign of the correlation in simple linear regression.
What is a good R-squared?
It depends on the field and the purpose. Values above 0.7 are often considered strong in social science, while some physical measurements routinely exceed 0.95. For prediction, held-out error is a better guide than any R² threshold.
Why is my intercept negative or impossible?
The intercept is the predicted y at x = 0, which may lie outside the data and outside physical plausibility. It anchors the line but is not necessarily a meaningful quantity. Center the predictor if you need an interpretable constant.
Does a significant slope prove causation?
No. A significant slope says the association is unlikely to be zero under the model, not that x causes y. Confounders, reverse causation, and selection effects can all produce a significant slope.
How many observations do I need?
Enough to estimate the residual variance and the slope precisely. A common rough guide is at least 10 to 20 observations per predictor, but a power calculation based on the smallest effect you care about is more defensible.
Further reading