Residual bootstrap regression holds the design fixed, resamples centered fitted errors and refits. Follow an exact example and see why scale, leverage and dependence matter.
Residual bootstrap
The residual bootstrap estimates uncertainty in a fitted regression by keeping the regressors fixed, resampling the fitted errors, adding them to the original fitted values, and fitting the model again. For a quant estimating a factor loading, it asks how much that coefficient might move if the same factor observations were accompanied by different errors drawn from an estimated common error distribution.
It is a model-based bootstrap: the fitted mean supplies the signal, and the empirical residual distribution supplies the noise. It does not require normally distributed errors. The basic pooled version does require a correctly specified linear conditional mean and independent, identically distributed errors with mean zero and finite variance. Homoskedasticity alone is weaker than that common-distribution assumption.
What stays fixed, and what is drawn
For a full-column-rank design matrix X, fit y=Xβ+ε by ordinary least squares. Calculate fitted values y^i=xi⊤β^ and residualse^i=yi−y^i. Center the residual pool:
e~i=e^i−e^ˉ,e^ˉ=n1i=1∑ne^i.
With an intercept, unadjusted OLS residuals already sum to zero. Without an intercept, do not assume that identity: centering explicitly makes the bootstrap error distribution have mean zero.
For each replicate, independently draw indices I1,…,In uniformly with replacement from 1,…,n, then construct
ei∗=e~Ii,yi∗=y^i+ei∗,β^∗=(X⊤X)−1X⊤y∗.
Refit every replicate. The spread of the refitted coefficients can estimate their standard errors; an interval needs its own construction, such as a basic bootstrap interval. Leaving the coefficients unchanged would discard the very variation being estimated. Shalizi's regression-bootstrap notes distinguish this fixed-design construction from resampling whole observations.
One complete replicate
Take the fictional, unitless observations x=(0,1,2) and y=(1,2,5), including an intercept. OLS gives
The residual mean is zero. Use the raw centered residuals, without a degrees-of-freedom or leverage adjustment, for this example. Suppose the sampled row indices are (2,1,1). Reusing row 1 is allowed because the draw is with replacement:
Fixed xi
Original y^i
Drawn ei∗
New yi∗
0
2/3
−2/3
0
1
8/3
1/3
3
2
14/3
1/3
5
For the new response, xˉ=1, yˉ∗=8/3, and the centered cross-product sum is 5. Thus
β^1∗=25=2.5,β^0∗=38−25=61.
This replicate moves the slope by 0.5. The original fitted values generate the new responses; the newly fitted line is the output.
Fictional unitless data share x values 0, 1 and 2. The original line has intercept 2/3 and slope 2. Drawing raw residuals −2/3, 1/3 and 1/3 gives responses 0, 3 and 5, with refitted intercept 1/6 and slope 2.5.
Figure: The design stays fixed while errors move between rows. Both panels mark the original fitted values; the right panel also shows the refitted bootstrap line.
The pool contains three entries but only two distinct values. A draw equals 1/3 with probability 2/3 and −2/3 with probability 1/3. Deduplicating the pool and drawing each distinct value with probability 1/2 would change the method. Nor is this a permutation: a bootstrap replicate need not use each residual once or have residual draws summing to zero.
Fitted errors are imperfect stand-ins
Even when the underlying errors are iid, estimated residuals are not exactly iid. Fitting imposes constraints and changes their variances. Under the homoskedastic linear model,
Var(e^i∣X)=σ2(1−hii),
where hii is leverage in regression. High-leverage rows have more strongly compressed residual variation. Shalizi, §18.3 derives the residual covariance matrix and this relationship.
Raw resampling also carries a finite-sample scale choice. Here its conditional error variance is
Var∗(ei∗)=3(1/3)2+(−2/3)2+(1/3)2=92.
The usual regression error-variance estimate instead divides the residual sum of squares by n−p, where p counts all fitted coefficients, including the intercept. Here n=3, p=2, so that estimate is 2/3. Multiplying the pool by n/(n−p) is one scale correction; leverage-based transformations are another choice. MacKinnon, §3.1 discusses these adjustments. They would change this replicate's numbers and are deliberately absent from the example.
Validity is a large-sample approximation under suitable design and error conditions, not an assertion that fitted residuals are observed independent errors. Fixed parameter dimension, a stable nondegenerate design and no observation dominating the fit are standard ingredients. Three observations demonstrate arithmetic; they cannot establish confidence-interval coverage. Increasing the number of bootstrap draws reduces simulation error but does not repair a misspecified mean or error model.
Choose the resampling unit to match the problem
A pairs bootstrap, a nonparametric bootstrap of entire (xi,yi) rows, changes the design as well as the response. It targets a different sampling experiment.
A wild bootstrap keeps residual scale attached to its row through random multipliers. It addresses heteroskedastic regression settings for which pooling raw residuals loses the variance pattern.
A block bootstrap resamples dependent stretches. Independent residual draws erase serial dependence, so they do not by themselves justify inference for a time-series factor regression.
The model and the sampling experiment determine which construction is defensible. A fixed-design residual bootstrap measures coefficient uncertainty conditional on the chosen design; it does not automatically include the noise of a future observation or uncertainty from a new set of factor realizations.
For the fitting identities used above, Simple Linear Regression: OLS Estimators and Geometry develops the slope formula and why intercept-model residuals sum to zero. The public preview is available; the full lesson requires membership.
