The short answer: A p-value is the probability of getting a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. A small p-value means the data are unusual under that assumption, not that the null hypothesis has a small probability of being true. It also says nothing about the size or practical importance of an effect.

Key takeaways

  • p is a conditional probability. It is calculated assuming the null hypothesis, not as a probability about the null hypothesis.
  • Smaller p means less compatibility with the null, not a larger or more important effect.
  • Alpha is a decision threshold chosen in advance. The 0.05 line is a convention, not a natural boundary.
  • Large samples make tiny differences significant. Always pair a p-value with an effect size and confidence interval.
  • Many tests inflate false positives. Twenty independent tests at α = 0.05 have about a 64% chance of at least one false alarm.

The logic of a significance test

A significance test starts with a null hypothesis, usually a statement of no difference or no effect: a filling machine dispenses 500 ml on average, a drug performs the same as placebo, a coin is fair. The alternative hypothesis says the parameter differs from that value, or is larger, or is smaller, depending on the question.

You then compute a test statistic from the sample and compare it with the distribution it would follow if the null were true. The p-value is the area in the tail, or tails, beyond the observed statistic. Extreme values are unlikely under the null, so a small tail area is evidence against it. The logic is indirect: the null is never proven, only judged more or less compatible with the data.

What the p-value is, precisely

For a one-sided test with a statistic T and observed value t, the p-value is P(T ≥ t | H0). For a two-sided test it is the probability of a result at least as far from the null in either direction. The vertical bar is the crucial part: the calculation conditions on the null hypothesis being true and on the model’s other assumptions.

Two-sided tests are the default when either direction would be interesting, because choosing a side after seeing the data doubles the chance of a false positive. One-sided tests are defensible when a result in the opposite direction would be treated as no effect, and the direction was specified before data collection.

Worked example: a filling machine

A machine is supposed to fill bottles to a mean of 500 ml with a standard deviation of 10 ml. A sample of 25 bottles averages 503 ml. The standard error is 10 ÷ √25 = 2 ml, so z = (503 − 500) ÷ 2 = 1.50. The upper-tail probability is about 0.0668, and the two-sided p-value is about 0.1336.

At α = 0.05 the result is not statistically significant. The sample mean is 3 ml above target, but a difference that large is not unusual when the standard error is 2 ml. Now suppose the sample had 100 bottles and the same 503 ml mean. The standard error becomes 1 ml, z = 3.00, and the two-sided p-value is about 0.0027. The estimated difference is identical; only the precision changed. That is why a p-value cannot be read as an effect size.

Alpha and the decision rule

The significance level α is the false-positive rate you are willing to tolerate before the data arrive. If p ≤ α, the result is called statistically significant and the null is rejected. If p > α, the null is not rejected — which is not the same as accepting it or proving it true.

The 0.05 convention is a convention. A result with p = 0.049 and a result with p = 0.051 are practically indistinguishable, yet the decision rule treats them as opposite. Report the exact p-value and let readers see how close the call was. In high-stakes screening, α = 0.001 may be more appropriate; in exploratory work, a looser threshold may be acceptable if the findings are labeled exploratory.

What a p-value is not

The most common error is reading p = 0.03 as “a 3% chance the null hypothesis is true.” That reverses the conditional probability. The p-value is P(data this extreme | null), not P(null | data this extreme). To get the latter you need a prior probability for the null, which the test does not supply.

The reversal matters. Suppose 100 of 1,000 hypotheses in a research program are true, the test has 80% power, and α = 0.05. Of the 100 true hypotheses, about 80 are detected. Of the 900 false ones, about 45 produce false positives. Among the 125 significant results, only 80 ÷ 125 = 64% are true, so 36% are false discoveries. A “significant” finding can still be more likely false than true when the prior odds are low.

p-values, effect size, and confidence intervals

A p-value combines the size of an effect and the precision of its estimate. That is why a huge study can report p < 0.001 for a difference too small to matter, and a small study can report p = 0.20 for a difference that would be important if real. The confidence interval separates those two ingredients: it shows both the plausible magnitude and the uncertainty.

Report the effect in the units of the problem. “The drug lowered blood pressure by 2 mmHg (95% CI, 1 to 3; p = 0.01)” is far more useful than “the result was significant.” A p-value answers whether the data are compatible with no effect; the interval answers how large the effect might be, which is usually the question people actually care about.

Multiplicity: many tests, many false alarms

If you run 20 independent tests at α = 0.05 when every null is true, the chance of at least one false positive is 1 − 0.95²⁰ ≈ 0.64. That arithmetic explains why subgroup fishing and large outcome batteries produce irreproducible “findings.” Bonferroni correction tests each hypothesis at α/m, which controls the family-wise error rate but is conservative when tests are correlated.

The false discovery rate is an alternative that controls the expected share of false positives among rejected hypotheses, which is often more suitable for screening thousands of genes or survey items. Either way, the correction must be planned, not chosen after seeing which results survive.

Common mistakes

  • Saying “there is a 5% chance the null is true” instead of “the data are unlikely under the null.”
  • Treating p > 0.05 as proof that there is no effect or that the groups are equivalent.
  • Calling a result “highly significant” because p is tiny, when the effect is trivial.
  • Changing the test, the outcome, or the subgroup after seeing the data.
  • Reporting only p-values and omitting effect sizes, confidence intervals, and sample sizes.
  • Comparing p-values across studies as if they measured the strength of evidence on a common scale.

Reporting p-values well

Give the exact p-value to two or three significant digits, the test statistic and degrees of freedom, the effect size with units, and a confidence interval. State whether the test was one- or two-sided and what the null hypothesis was. If multiple tests were run, say how they were adjusted, and distinguish pre-specified from exploratory analyses.

Finally, write conclusions in the language the design supports. “The data are compatible with a small positive effect” is honest. “The treatment works” is a causal claim that a p-value alone cannot license.

Frequently asked questions

What does p = 0.05 mean?

It means that, if the null hypothesis and model assumptions are true, there is about a 5% chance of a test statistic at least as extreme as the one observed. It does not mean there is a 5% chance the null is true.

Is a smaller p-value stronger evidence?

Within a single pre-specified test, a smaller p means the observed result is less likely under the null. Across different studies, sample sizes, and models, p-values are not comparable measures of evidence.

Can a result be significant but unimportant?

Yes. With a large enough sample, a tiny difference can produce a very small p-value. That is why effect size and confidence intervals belong next to every p-value.

What if p is exactly 0.05?

The decision is arbitrary at the boundary. Report the exact value, note that it sits on the threshold, and interpret it alongside the interval rather than treating the threshold as a cliff edge.

Why do some studies use 0.01 or 0.001?

Stricter thresholds reduce false positives when many tests are run or when a false claim is costly. The choice should be justified by the consequences of each type of error, not by convention alone.

Is a p-value the same as a confidence interval?

No, but they are linked. A 95% confidence interval that excludes the null value corresponds to a two-sided test with p < 0.05. The interval adds the plausible range of effect sizes.

Further reading

Keep exploring

Read next Logarithm: Base, Product Rule, and Scale A logarithm is an exponent in disguise. Learn the definition, the product, quotient and power rules, change of… Read next Order of Operations: PEMDAS and Ambiguous Expressions PEMDAS is a shared convention, not a law of mathematics. Learn the real precedence rules, the exponent and… Read next Significant Figures: Counting, Rounding, and Precision Significant figures record how precisely a value was measured. Learn the counting rules, the addition and multiplication rules,…