Skip to content

Group sequential testing and alpha spending

Control false rejections across interim looks: cumulative alpha spending, information fractions and why repeated 5% tests exceed a 5% total error budget.

Group sequential testing evaluates an accumulating dataset at a sequence of interim analyses, using stopping boundaries calibrated for the whole monitoring procedure. Alpha spending specifies how much of the total Type I error probability may have been used by each information fraction. For a quant testing a mean-return claim as new observations arrive, the practical issue is that two opportunities to declare significance create more false-rejection risk than one.
A monitoring design can stop early when evidence crosses a boundary. Its error guarantee concerns the chance of ever rejecting the true null, including an early rejection that would disappear in a later sample. It does not attach a fresh error allowance to every review.

Two looks at the same research claim

Consider a fictional, tightly specified experiment: independent normal observations with a common unknown mean and a known, constant variance. Test the null hypothesis that the mean is zero, against a two-sided alternative, after the first batch and after an equally large second batch. These are the only two permitted looks in this example.

Under the null, write the standardised statistics as

Z1∼N(0,1),Z2=Z1+Znew2,Z_1\sim\mathcal{N}(0,1),\qquad Z_2=\frac{Z_1+Z_{\mathrm{new}}}{\sqrt{2}},

where Znew∼N(0,1)Z_{\mathrm{new}}\sim\mathcal{N}(0,1) is independent of Z1Z_1. Both looks have standard normal marginal distributions, but their correlation is 1/2≈0.70711/\sqrt{2}\approx0.7071: the second analysis includes the first batch.

If either ∣Z1∣\lvert Z_1\rvert or ∣Z2∣\lvert Z_2\rvert exceeds Φ−1(0.975)≈1.959964\Phi^{-1}(0.975)\approx1.959964, reject. Here Φ\Phi is the standard normal cumulative distribution function. Although each fixed-look test has a 5% rejection probability under the null, the whole procedure has

P0 ⁣(∣Z1∣>1.959964  or  ∣Z2∣>1.959964)≈0.083118.\mathbb{P}_0\!\left(\lvert Z_1\rvert>1.959964 \;\text{or}\;\lvert Z_2\rvert>1.959964\right) \approx0.083118.
That is 8.31%, calculated with the unrounded quantile. The independent-tests formula 1−0.9521-0.95^2 does not apply because the looks overlap. Positive correlation does not make the second chance free. The gsDesign theory manual develops this joint normal model and its repeated-testing problem.

A simple safe budget, with a price

Allocate a two-sided marginal budget of 2.5% to each look. The cutoff is then
c=Φ−1(1−0.025/2)≈2.241403.c=\Phi^{-1}(1-0.025/2)\approx2.241403.

For rejection events Ri={∣Zi∣>c}R_i=\{\lvert Z_i\rvert>c\}, the Bonferroni inequality gives

P0(R1∪R2)≤P0(R1)+P0(R2)=0.025+0.025=0.05.\mathbb{P}_0(R_1\cup R_2) \le\mathbb{P}_0(R_1)+\mathbb{P}_0(R_2) =0.025+0.025=0.05.
This bound needs valid marginal tests, but no independence assumption. In this correlated normal example, the actual probability is 4.280812%, below the 5% cap. The second look adds only 1.780812 percentage points of new false-rejection probability, because some paths exceeding its cutoff have already rejected at look 1.
This is a conservative per-look allocation. It is not an exact Lan–DeMets spending boundary. Spending increments describe first crossings among paths still being monitored; marginal tail areas also count paths that crossed earlier.
In the correlated two-look normal model, reusing 1.960 gives cumulative null rejection probabilities 5% then 8.31%. A 2.241 cutoff at both looks gives 2.5% then 4.28%. Linear cumulative spending calibrated jointly uses 2.241 then 2.125 and reaches 5%.
In the correlated two-look normal model, reusing 1.960 gives cumulative null rejection probabilities 5% then 8.31%. A 2.241 cutoff at both looks gives 2.5% then 4.28%. Linear cumulative spending calibrated jointly uses 2.241 then 2.125 and reaches 5%.
Original exact-model numerical integration, not simulation or market data. Markers show monitored looks; connecting lines guide the eye. Absolute Z cutoffs appear in the table.

The connecting lines do not authorise testing between the two monitored looks.

Spending is cumulative; boundaries are calibrated jointly

Let IiI_i be statistical information at look ii and Imax⁡I_{\max} the planned final information. Define the information fraction
ti=IiImax⁡.t_i=\frac{I_i}{I_{\max}}.

For the known-variance mean model, information is ni/σ2n_i/\sigma^2, so the two looks occur at t1=1/2t_1=1/2 and t2=1t_2=1. Information fractions need not equal fractions of calendar time. Unequal precision or dependent observations can invalidate a sample-count shortcut and the canonical covariance formula.

A cumulative spending function A(t)A(t) is nondecreasing, with A(0)=0A(0)=0 and A(1)=αA(1)=\alpha. Choose it before examining the evidence. In a symmetric two-sided design without a separate futility boundary, calibration seeks cutoffs satisfying

P0 ⁣(⋃j=1i{∣Zj∣>cj})=A(ti).\mathbb{P}_0\!\left(\bigcup_{j=1}^{i}\{\lvert Z_j\rvert>c_j\}\right)=A(t_i).

Equivalently, the new error probability at look ii is

P0 ⁣(∣Zj∣≤cj for all j<i,  ∣Zi∣>ci)=A(ti)−A(ti−1),\mathbb{P}_0\!\left( \lvert Z_j\rvert\le c_j\text{ for all }j<i, \;\lvert Z_i\rvert>c_i\right) =A(t_i)-A(t_{i-1}),
with t0=0t_0=0. This is an unconditional probability of first rejection at that look, not a rejection probability conditional on reaching it. Finding cic_i requires the joint distribution of the looks, not just their individual normal distributions. In the canonical independent-increment model,
Cov(Zi,Zj)=Ii/Ij,i<j.\text{Cov}(Z_i,Z_j)=\sqrt{I_i/I_j},\qquad i<j.
For an original numerical illustration, take A(t)=0.05tA(t)=0.05t in the same two-look model. The first cutoff remains c1=2.241403c_1=2.241403, spending 2.5%. Solving the first-crossing equation for the remaining 2.5% gives c2≈2.125119c_2\approx2.125119. Joint numerical integration then gives a total error probability of 5%, to numerical precision. The second cutoff can be lower than the conservative budget's 2.241403 because its new error allowance excludes paths already rejected.
Lan and DeMets (1983), §§2–3, construct boundaries from cumulative crossing probabilities and explain why current and past analysis times suffice. The spending-function manual, §8.1, makes the link between information fractions and cumulative spending explicit. These are design equations, not a rule to compare an ordinary pp-value with A(ti)A(t_i).

What the monitoring plan must settle

Specify the hypothesis, sidedness, statistic, information measure, spending rule and stopping decisions before monitoring. Choose the final information target together with the alternative effect and desired statistical power. A conservative error bound alone does not establish useful power or an appropriate sample size.
The fixed two-look allocation above covers exactly those looks. Adding a third review changes its rejection event and requires a new valid design. A Lan–DeMets spending approach can accommodate changed analysis timing or additional looks through its prespecified spending rule and recalculated boundaries, under the required joint model and timing conditions. Choosing an extra look because an unblinded statistic is nearly significant is not automatically covered. Separate futility rules also need a stated treatment: whether a futility stop must be obeyed can affect the error calibration, as the boundary-derivation manual, §6.1.1 explains.
Repeated monitoring of one claim is also distinct from testing many strategies or outcomes. Spending across time does not by itself control errors across that larger family. Nor does it repair look-ahead leakage, serial dependence, or a misspecified return model. The Confidence Intervals and Hypothesis Testing lesson develops the fixed-sample mean test and its sampling assumptions; the public preview is open and the full lesson requires membership.
For monitoring at arbitrary stopping times, examine a method with an explicit time-uniform guarantee, such as a valid e-process or confidence sequence. That is a separate construction, with its own assumptions. A finite list of valid group-sequential boundaries does not acquire such a guarantee merely by being labelled “sequential.” Howard et al. define confidence sequences by simultaneous validity over an unbounded time horizon.

Sources

Last updated
  • sequential testing
  • alpha spending
  • hypothesis testing
  • type i error
  • research design
Keep connecting the ideas
← Back to the glossaryBack to top ↑