Tukey HSD test: Definition & Examples

The Tukey HSD test—Tukey's honestly significant difference test—compares every pair of population means while controlling the chance of at least one false rejection across that family. For a quant comparing mean execution costs across several venues, it answers which gaps are statistically detectable after accounting for the whole set of pairwise comparisons.

An omnibus analysis of variance asks whether all means are equal. It does not identify the pairs responsible for a difference. Tukey HSD is commonly used for that follow-up question; its adjustment is designed specifically for all pairwise mean comparisons. NIST explains the simultaneous confidence guarantee.

One threshold for a balanced design

Suppose there are

k\ge2

fixed groups, each containing

n\ge2

observations. The classical model assumes observations are independent within and across groups, each group's errors are normally distributed, and all groups share a common positive, finite error variance. Their population means may differ.

Let $\bar y_i$ be group $i$ 's sample mean. Estimate the common variance using the pooled within-group mean square:

\mathrm{MSE}=\frac{\sum_{i=1}^{k}\sum_{r=1}^{n}(y_{ir}-\bar y_i)^2}{\nu}, \qquad \nu=k(n-1).

Write

q_{1-\alpha;k,\nu}

for the lower-tail $1-\alpha$ quantile of the studentised range distribution. The honestly significant difference is

\mathrm{HSD}=q_{1-\alpha;k,\nu}\sqrt{\frac{\mathrm{MSE}}{n}}.

Reject the pairwise null hypothesis

\mu_i=\mu_j

when

|\bar y_i-\bar y_j|>\mathrm{HSD}

. Equivalently, check whether zero falls outside the interval

(\bar y_i-\bar y_j)\pm\mathrm{HSD}.

These are simultaneous confidence intervals: under the balanced normal model, the entire collection covers its true pairwise differences with probability

1-\alpha

over repeated samples. The studentised range calibrates the largest standardised spread, so the threshold accounts for inspecting every pair. The denominator here is

\sqrt{\mathrm{MSE}/n}

; inserting the ordinary two-mean standard error

\sqrt{2\mathrm{MSE}/n}

would add an erroneous factor of

\sqrt{2}

. Abdi and Williams give both balanced and unequal-size formulas.

Three venues, only one detected gap

Consider a stipulated teaching example of execution cost in basis points (bp), where a larger value means a higher cost. Venues A, B and C have five observations each and sample means of 10, 12 and 15 bp. Stipulate pooled

\mathrm{MSE}=4\ \mathrm{bp}^2

and the model assumptions above.

There are $15-3=12$ residual degrees of freedom. At family level $\alpha=0.05$ ,

q_{0.95;3,12}\approx3.772929, \qquad \mathrm{HSD}=3.772929\sqrt{4/5}\approx3.374610\ \mathrm{bp}.

The quantile is computed using the studentised range distribution's inverse CDF.

Directed comparison	Observed gap (bp)	95% simultaneous interval (bp)	Reject equal means?
B minus A	2	$[-1.375,\ 5.375]$	No
C minus B	3	$[-0.375,\ 6.375]$	No
C minus A	5	$[1.625,\ 8.375]$	Yes

Stipulated three-venue example: B minus A is 2 bp with interval minus 1.375 to 5.375; C minus B is 3 bp with interval minus 0.375 to 6.375; C minus A is 5 bp with interval 1.625 to 8.375. Only C minus A excludes zero.

Original fixed-input illustration. All three intervals use the same 3.3746 bp margin and jointly target 95% coverage under the stated model.

Only A versus C clears the threshold. The other two nonrejections do not establish equality, and they do not imply that A and C must have equal means. A statistically detected cost gap also does not determine which venue is preferable after fees, fill quality and operational constraints.

These summary inputs support the arithmetic, not evidence that a real dataset satisfies the assumptions. No raw observations or sampling design are supplied. In an actual venue study, shared market shocks, repeated parent orders and changing volatility can undermine independence or a common variance; checking a formula cannot resolve those design issues.

What the adjustment protects

The family-wise error rate is the probability of one or more false rejections within the defined family. Tukey's procedure controls it at no more than $\alpha$ under its model. This differs from the false discovery rate, which concerns the expected proportion of false rejections among all rejections, with that proportion defined as zero when nothing is rejected.

Running each pair's t-test at an unadjusted 5% level does not generally provide a 5% family guarantee. Nor does Tukey's adjustment for three declared venues automatically cover an earlier search over many venue definitions, time windows or performance measures. Define the comparison family and analysis before using the result as evidence.

Unequal group sizes need Tukey–Kramer

With unequal

n_i

, the common-variance extension uses a pair-specific margin:

q_{1-\alpha;k,N-k} \sqrt{\frac{\mathrm{MSE}}{2}\left(\frac{1}{n_i}+\frac{1}{n_j}\right)}, \qquad N=\sum_{i=1}^{k}n_i.

This is the Tukey–Kramer method; substituting one convenient sample size into the balanced formula is incorrect. The expression reduces to HSD when both sizes equal

n

. Unequal sample sizes do not remove the common-variance assumption. Unequal-variance alternatives such as Games–Howell use different uncertainty calculations; SciPy's documentation distinguishes these procedures and states their assumptions.

For a full derivation of the pooled variance and the global test, read One-Way Analysis of Variance (public preview; full lesson requires membership).

Sources

NIST/SEMATECH, §7.4.7.1: Tukey's method—all-pairs confidence coverage, studentised range and unequal-size distinction.
Abdi and Williams (2010), Tukey's Honestly Significant Difference Test, §§1–3—pairwise comparison setup and threshold formulas.
SciPy: tukey_hsd, Notes and Parameters—independence, normality, equal variance and method distinctions.
SciPy: studentized_range, Methods and Notes—quantile convention and distribution parameters.

One threshold for a balanced design

Three venues, only one detected gap

What the adjustment protects

Unequal group sizes need Tukey–Kramer

Sources

Related terms

Normal distribution

Null hypothesis

Student's t-test