TOST tests whether a difference lies inside predeclared practical margins. Work through two one-sided z tests, the matching 90% interval, and why nonsignificance does not establish equivalence.
Two one-sided equivalence test
A two one-sided equivalence test (TOST) tests whether a population difference lies strictly between two predeclared bounds of practical importance. A quant comparing a new execution method with a reference method might ask whether their mean performance differs by less than a specified tolerance in either direction. This asks for evidence of a sufficiently small difference; an ordinary hypothesis test that fails to detect a difference from zero does not supply that evidence.
Put the unacceptable differences in the null
Let δ denote the population mean difference, measured as new minus reference, and choose L<U before inspecting the results. The hypotheses are
H0:δ≤Lorδ≥U,H1:L<δ<U.
TOST rejects the lower null δ≤L and the upper null δ≥U separately, requiring both rejections. Each component uses the same significance level α. Schuirmann's original paper describes this construction and its connection to the central 100(1−2α)% interval. Schuirmann (1987), pp. 660–661.
There is no Bonferroni division to α/2 here. This is an intersection–union test: under the overall null, at least one component null is true. Rejecting both necessarily rejects that true component, whose rejection probability is at most α. Therefore the overall false-equivalence probability is at most α, provided the component tests are valid. This argument does not require their independence.
The margins express the economic question. A tolerance in basis points must specify the performance measure, horizon and cost treatment. Widening it after seeing the estimate changes the question. Equivalence also does not assert exact equality or equal tail risk. Lakens discusses predeclared bounds and distinguishes equivalence from nonsignificance. Lakens (2017), pp. 355–356.
A known-standard-error calculation
Consider an original hypothetical comparison of mean performance per trade, after the same specified transaction costs. Declare the acceptable region to be (−1,+1)basis points per trade. These are illustrative tolerances, not recommended thresholds.
Stipulate a normally distributed estimator δ^ with mean δ and knownstandard errorse=0.3 bp. Observe δ^=0.2 bp and use α=0.05. With Φ denoting the standard normal CDF:
Component null
Statistic
One-sided p-value
δ≤−1
zL=(0.2+1)/0.3=4
1−Φ(4)=0.00003167
δ≥+1
zU=(0.2−1)/0.3=−8/3
Φ(−8/3)=0.00383038
Both p-values are below 0.05, so TOST establishes equivalence within the stipulated margins under this model. The overall TOST p-value is their maximum, approximately 0.00383.
Its endpoints lie strictly inside (−1,+1). Algebraically, its lower endpoint exceeds L exactly when zL>z1−α; its upper endpoint is below U exactly when zU<−z1−α. This article uses strict rejection, p<α: an interval endpoint exactly on a margin is insufficient. Equality conventions have probability zero in this continuous model, but rounded displayed endpoints must not decide a boundary case.
Two hypothetical normal estimates both equal 0.2 bp. With known SE 0.3 bp the 90% interval is −0.293 to 0.693 bp, strictly inside the −1 to +1 bp margins. With known SE 0.8 bp the interval is −1.116 to 1.516 bp and crosses both margins. Ordinary two-sided zero-test p-values are 0.505 and 0.803 respectively.
Original known-SE examples. Dots show the same estimate; interval width determines whether the chosen equivalence margins can be ruled out.
What the zero test misses
For the main example, the ordinary two-sided test of δ=0 gives z=0.2/0.3=2/3 and p≈0.505. With the same estimate but known SE 0.8 bp, that test also fails to reject (p≈0.803), while the 90% interval now crosses both margins. Equivalence is not established in this second case. Neither result proves the null of its own test.
Statistical difference and practical equivalence can also coexist: a precisely estimated, small nonzero difference can exclude zero while remaining inside the margins. A 95% interval answers the usual 5% two-sided zero-test question; substituting it for the 90% interval makes this TOST criterion more conservative.
Match uncertainty to the design
The calculation above assumes a known sampling standard error. With estimated variance in a normal mean design, use the appropriate Student t reference distribution and degrees of freedom; independent unequal-variance samples and paired observations require different uncertainty calculations. Lakens supplies the independent and dependent-mean versions. Lakens (2017), pp. 357–358.
Paired trade comparisons, serial dependence and repeated searches across methods must be reflected in the inference design. Merely inserting a sample standard deviation into the known-SE calculation does not preserve its guarantees. The Confidence Intervals and Hypothesis Testing lesson develops the normal-family t pivot and interval/test relationship (public preview; full lesson requires membership). NIST also explains why confidence levels describe repeated-sampling coverage rather than a posterior probability that the fixed difference lies in the observed interval. NIST, §1.3.5.2.
