Key takeaways
- z-tests need a known σ or a large sample. The standard normal is the reference distribution only under that condition.
- The denominator is the standard error, not the standard deviation of the raw data.
- Two-tailed and one-tailed p-values differ by a factor of two when the test statistic is fixed.
- A p-value is not the probability that the null hypothesis is true.
- Statistical significance is not practical importance. Report an effect size and a confidence interval.
- Peeking at data and stopping when p < 0.05 inflates the false-positive rate.
What a z-test does
A z-test is a hypothesis test for a mean or a difference of means that uses the standard normal distribution as its reference. The null hypothesis specifies a value, such as μ = 100 or μ₁ − μ₂ = 0. The test statistic measures how many standard errors the observed estimate sits from that value, and the p-value is the probability of a result at least that extreme if the null hypothesis were true.
The test is appropriate when the population standard deviation is known, or when the sample is large enough that the sample standard deviation is a reliable substitute. It assumes independent observations and a roughly normal sampling distribution, which the central limit theorem supplies for large samples even when the raw data are skewed.
One-sample z-test
For a sample mean x̄ from n independent observations with known standard deviation σ:
z = (x̄ − μ₀) ÷ (σ ÷ √n)
Standard error = σ ÷ √n
Suppose a machine is specified to fill bottles with a mean of 100 mL and σ = 15 mL is known from long production records. A sample of 36 bottles averages 104 mL. The standard error is 15 ÷ 6 = 2.5, so z = (104 − 100) ÷ 2.5 = 1.60. The sample mean is 1.6 standard errors above the target.
For a two-tailed test at α = 0.05, the critical values are ±1.96. Since 1.60 is inside that range, the result is not statistically significant at the 5% level, and the two-tailed p-value is 0.1096.
Two-sample z-test
For two independent samples with known standard deviations:
z = (x̄₁ − x̄₂) ÷ √(σ₁²/n₁ + σ₂²/n₂)
Group 1 has x̄₁ = 52, σ₁ = 10, n₁ = 40. Group 2 has x̄₂ = 48, σ₂ = 12, n₂ = 50. The standard error is √(100/40 + 144/50) = √(2.5 + 2.88) = √5.38 = 2.319. The difference is 4, so z = 4 ÷ 2.319 = 1.725.
The two-tailed p-value is 0.0846, above 0.05, so the null hypothesis of equal means is not rejected at that level. The 95% confidence interval for the difference is 4 ± 1.96 × 2.319, which runs from −0.55 to 8.55 and includes zero, consistent with the test decision.
One-tailed versus two-tailed
A two-tailed test asks whether the mean differs in either direction, so it splits α across both tails. A one-tailed test asks whether the mean is larger (or smaller) in a specified direction and puts all of α in one tail. For the one-sample example, the upper-tailed p-value is 0.0548, half the two-tailed value, and the critical value at α = 0.05 is 1.645. The result is still not significant at 5%, but it is close.
Choose the direction before seeing the data. Switching from two-tailed to one-tailed after the fact is a form of p-hacking. Common two-tailed critical values are 1.645 at 90%, 1.960 at 95%, and 2.576 at 99% confidence.
P-values and confidence intervals
The p-value is the probability, computed under the null hypothesis, of observing a test statistic at least as extreme as the one obtained. It is not the probability that the null is true, and 1 − p is not the probability that the alternative is true. A p-value of 0.03 means that a result this extreme would occur about 3% of the time if the null held, not that there is a 97% chance the effect is real.
A confidence interval is usually more informative because it shows both the estimate and its uncertainty. The 95% interval for the bottle mean is 104 ± 1.96 × 2.5, or 99.1 to 108.9 mL. It includes the target 100, matching the failure to reject. Report the interval, the sample size, and the effect size together.
Assumptions and diagnostics
Observations must be independent. Repeated measurements on the same person, clustered sampling by classroom or clinic, and time-series data violate this and require models that account for the dependence. Samples must be drawn without bias, and the two-sample version assumes the groups are independent of each other.
The known-σ condition is strict. In practice σ is almost always estimated from the sample, which makes the t distribution the correct reference for small n. For large n the t distribution approaches the standard normal, so the z-test is a reasonable large-sample approximation. Check for extreme outliers, because a single wild value can move the mean and the standard deviation substantially.
When to use a t-test instead
Use a t-test when σ is unknown and estimated by s, especially when n is below about 30. The one-sample t statistic is t = (x̄ − μ₀) ÷ (s ÷ √n) with n − 1 degrees of freedom. The two-sample version has a Welch form that does not assume equal variances and is the safer default.
For the bottle example with s = 15 and n = 36, t = 1.60 and the two-tailed p-value is about 0.119, close to the z result because the sample is moderate. With n = 8 the same t statistic would have a much heavier tail and a larger p-value. Using z with an estimated σ in a small sample understates the p-value.
Effect size, power, and multiple testing
A significant result can be trivial. Cohen’s d = (x̄ − μ₀) ÷ σ = 4 ÷ 15 = 0.27 in the bottle example, a small effect. With a large enough sample, even a 0.1 mL deviation can produce a tiny p-value, so practical relevance must be judged against the process, not the p-value alone.
Power is the probability of detecting an effect that is really there. For a two-sided test at α = 0.05 and 80% power, the required sample size is approximately n = (1.96 + 0.84)² × (σ ÷ Δ)². To detect a 5 mL shift with σ = 15, n = 7.85 × 9 ≈ 71 bottles. Testing many hypotheses without adjustment also inflates false positives; Bonferroni’s α ÷ m is a simple, conservative correction.
Common mistakes
- Using s in place of σ with a small sample. Switch to a t-test.
- Dividing by σ instead of σ ÷ √n. The standard error shrinks as the sample grows.
- Doubling a one-tailed p-value when the direction was pre-specified, or halving a two-tailed p-value after seeing the data.
- Interpreting p as P(H₀ is true). It is P(data this extreme | H₀).
- Saying “the null is accepted” instead of “not rejected.” Absence of evidence is not evidence of absence.
- Ignoring dependence or clustering, which makes the standard error too small.
Practical caveats
The z-test is a large-sample tool, and “large” depends on the shape of the data. A sample of 40 is large for roughly symmetric measurements but may be inadequate for heavily skewed income or count data. When the sampling distribution is not close to normal, a bootstrap or a transformation may be more trustworthy.
Statistical software often defaults to two-tailed tests and pooled or Welch variances without saying so. Report which test, which tail, and which standard deviation you used. If the result matters for a decision, pre-register the analysis and avoid stopping early when the p-value crosses a threshold.
Frequently asked questions
What is the difference between a z-test and a t-test?
A z-test uses the standard normal distribution and assumes the population standard deviation is known. A t-test estimates the standard deviation from the sample and uses heavier tails, which matters most for small samples.
Can I use a z-test with a sample of 10?
Only if σ is genuinely known and the data are roughly normal. If σ is estimated from those 10 observations, use a t-test instead.
What does a p-value of 0.05 mean?
If the null hypothesis were true, results at least as extreme as the observed one would occur about 5% of the time. It is a statement about the data under a model, not about the probability that the null is true.
Why did my two-tailed and one-tailed tests disagree?
They answer different questions and use different critical regions. A result can be significant in a one-tailed test and not in a two-tailed test at the same α. The direction must be justified before collecting data.
Do I need equal sample sizes for a two-sample z-test?
No. The formula uses each group’s own n and σ. Equal sizes improve power slightly when variances are similar but are not required.
What if my data are not normally distributed?
For large samples the central limit theorem usually makes the sample mean approximately normal. For small, skewed samples, use a t-test with caution, transform the data, or use a bootstrap or nonparametric test.
Further reading