High-dimensional mean test

Why the inverse becomes the obstacle

With sample sizes $n_1,n_2$, the pooled covariance has rank at most $n_1+n_2-2$: centring removes one independent row from each group. If $p>n_1+n_2-2$, inversion is impossible. For example, 120 observations in one group and 100 in the other can give rank at most 218, insufficient for 500 coordinates. Even before that boundary, nearly redundant coordinates can make inversion sensitive to small data changes.

A pseudoinverse does not restore the classical reference law. It selects a different distance, with its own inference problem.

Estimate the squared mean gap directly

Let $\mathbf X_1,\ldots,\mathbf X_{n_1}$ and $\mathbf Y_1,\ldots,\mathbf Y_{n_2}$ be independent samples, with $n_1,n_2\ge2$, each i.i.d. within its group, with means $\boldsymbol\mu_X,\boldsymbol\mu_Y$ and finite covariance matrices $\Sigma_X,\Sigma_Y$. Coordinates within a row may be correlated; the two covariance matrices need not be equal. Test

The raw Chen–Qin statistic is

An equivalent identity explains the noise correction. Let $S_X,S_Y$ be the sample covariances with denominators $n_1-1,n_2-1$. Then

A tiny example, with no p-value attached

Take fictional pricing-error vectors in basis points:

Here $n_1=n_2=3$, $p=2$, $\bar{\mathbf X}=(2,1)^\top$ and $\bar{\mathbf Y}=(1/3,1/3)^\top$. The six ordered off-diagonal dot products in $X$ sum to $26$; those in $Y$ sum to zero. The cross-group term is $2\bar{\mathbf X}^\top\bar{\mathbf Y}=2$. Hence

Independently, the squared sample-mean gap is $29/9$, while $\operatorname{tr}(S_X)=2$ and $\operatorname{tr}(S_Y)=2/3$. The identity gives $29/9-2/3-2/9=7/3$ again.

Sampling noise contributes to the observed squared gap. Subtracting its estimated contribution gives the raw Chen–Qin U statistic; the tiny example demonstrates arithmetic, not high-dimensional inference.

Turning the estimate into a test

Under $H_0$, the variance is

Their Theorems 1–2 require growing samples and dimension, balanced sample-size proportions, a linear latent-variable model with moment restrictions, and covariance trace conditions. For equal covariances, a key restriction is

A dominant covariance direction can violate it. Their local-alternative regime requires, writing $\boldsymbol\delta=\boldsymbol\mu_X-\boldsymbol\mu_Y$ and $N=n_1+n_2$,

What the result would mean on a desk

The target is a joint mean difference, not equality of whole distributions. A rejection does not identify the affected instruments. Subsequent coordinate searches create a multiple-testing problem.

The Euclidean target also makes units consequential: changing one coordinate from basis points to percent changes its weight. Any prespecified rescaling defines a new target and must fit the inference assumptions. Adding unrelated coordinates adds noise; the number of columns is not a measure of usable evidence.

Daily errors can be serially dependent, and running two models on the same dates creates paired observations. Neither design satisfies the independent two-sample setup above. The relevant sampling variance and calibration must match the actual design.

Sources

• Chen and Qin (2010), A two-sample test for high-dimensional data with applications to gene-set testing, Annals of Statistics 38(2), 808–835 — Sections 2–3, raw statistic, assumptions, variance estimation, testing and the separate paired-observation construction.
• Jonathan Taylor, Stanford STATS305C, Hotelling T2 — classical two-sample covariance scaling and the need to calibrate alternative statistics separately.