Iteratively reweighted least squares: Definition & Examples

Iteratively reweighted least squares (IRLS) is a fitting algorithm that repeatedly solves a weighted least-squares problem, recalculating the weights from the current fit. For a quant fitting a response model to noisy financial observations, it can make a nonquadratic fitting criterion computationally accessible. The objective determines what the weights mean: IRLS is an algorithmic pattern, not a guarantee of robustness.

A single weighted least-squares fit treats its supplied weights as fixed. IRLS adds the feedback loop: fit → recompute weights → refit, until a stated stopping rule is satisfied.

Turning a robust score into a weighted fit

For robust linear M-estimation, consider the fixed-scale objective

L(\boldsymbol\beta)=\sum_{i=1}^n\rho(u_i), \qquad u_i=\frac{y_i-\mathbf x_i^\top\boldsymbol\beta}{s}, \qquad s>0.

Here

s

is a fixed residual scale,

\rho

is the loss, and

\psi=\rho'

is its score function. Differentiating gives the estimating equation

\sum_{i=1}^n\mathbf x_i\psi(u_i)=\mathbf0.

Writing $\psi(u)=w(u)u$ suggests the iteration

\begin{aligned} u_i^{(t)}&=\frac{y_i-\mathbf x_i^\top\boldsymbol\beta^{(t)}}s, & w_i^{(t)}&=\frac{\psi(u_i^{(t)})}{u_i^{(t)}},\\ \boldsymbol\beta^{(t+1)} &\in\operatorname*{arg\,min}_{\boldsymbol\beta} \sum_{i=1}^n w_i^{(t)}(y_i-\mathbf x_i^\top\boldsymbol\beta)^2. \end{aligned}

The weights stay fixed inside each subproblem, then change before the next. This construction requires suitable nonnegative weights; it is not valid for an arbitrary loss without checking its weight function. At a zero residual, use the continuous limit of

\psi(u)/u

when it exists. The MASS documentation explicitly distinguishes the weight ratio from the score derivative; its implementation recomputes residual weights and solves a weighted regression.

For Huber loss with threshold $k>0$ ,

\psi_k(u)= \begin{cases} u,&\lvert u\rvert\le k,\\ k\operatorname{sign}(u),&\lvert u\rvert>k, \end{cases} \qquad w_k(u)= \begin{cases} 1,&\lvert u\rvert\le k,\\ k/\lvert u\rvert,&\lvert u\rvert>k. \end{cases}

Thus

w_k(0)=1

. Large residuals get smaller weights but are not deleted. The clipped score caps each observation's residual contribution; in regression, multiplication by

\mathbf x_i

means this alone does not protect against extreme leverage. SAS's M-estimation documentation makes that limitation explicit.

A weighted mean that needs another iteration

Take synthetic observations

0,0,0,10

, in standardised units. Fit only a location

m

, with known fixed scale $s=1$ , Huber threshold

k=1.5

and starting value

m^{(0)}=0

. These choices define the example; they are not software defaults or a return model.

The initial residuals are $0,0,0,10$ , so the initial weights are $1,1,1,0.15$ . A location-only weighted least-squares fit is the weighted mean:

m^{(1)}=\frac{1(0)+1(0)+1(0)+0.15(10)}{1+1+1+0.15} =\frac{10}{21}\approx0.476190.

That is the first update, not the final estimate. Recomputing the weights gives another update of

210/421\approx0.498812

The converged location is $m=0.5$ . At that point the three zero observations each have residual $-0.5$ and score $-0.5$ ; the observation at 10 has residual $9.5$ and score $1.5$ . Hence

\sum_{i=1}^4\psi_{1.5}(y_i-0.5) =3(-0.5)+1.5=0.

The final outlier weight is

1.5/9.5=3/19\approx0.157895

. It differs from the initial

0.15

, and the weighted mean using those final weights is exactly

0.5

. Ordinary least squares would give the sample mean

2.5

. The robust fit remains above zero: down-weighting does not automatically remove contamination bias or recover a clean population mean.

Synthetic observations 0, 0, 0, 10 with fixed scale 1 and Huber threshold 1.5 receive initial weights 1, 1, 1, 0.15. Starting at location 0, IRLS first updates to 0.476190 and approaches the score root 0.5. The outlier's final weight is 0.157895.

Original synthetic example. Initial robust weights are shown beside the first five updates; the dashed line marks the converged location. All residuals use the same fixed scale.

What convergence does—and does not—establish

For this fixed-scale Huber example, the loss is convex and the score root is the unique global minimum. In more general regression, convexity gives global optimality of a minimiser; uniqueness needs additional conditions on the loss and design. Full design rank makes each positive-weight least-squares subproblem well defined, but is not a universal uniqueness proof for the original robust objective.

Redescending scores, such as Tukey's bisquare, can produce multiple local minima. Initialisation matters, and zero weights can reduce effective rank. The MASS documentation's Details section contrasts Huber's convex objective with the local minima of other scores. Updating the scale also changes the procedure; the fixed-scale derivation above does not establish convergence of a joint coefficient-and-scale algorithm.

A useful stopping check examines parameter changes and the original score or objective, with an iteration cap. A small change in one vector is a numerical criterion, not a general theorem of global optimality. SAS describes several convergence criteria, including a separate gradient check. Nor should the last weighted fit automatically inherit ordinary fixed-weight regression standard errors: its weights were estimated from the responses.

The other common IRLS: GLM fitting

A generalised linear model can also be fitted by IRLS. There the algorithm updates a working response as well as working weights determined by the link derivative and model variance, as in the R fitting implementation. These weights serve likelihood or quasi-likelihood fitting; they do not automatically down-weight outliers in the robust M-estimation sense. R's official glm documentation identifies IWLS as its default fitting method and distinguishes prior weights from the final working weights.

For the separate inverse-variance interpretation of WLS, SelfMadeQuant's Generalised Least Squares derives the transformed regression and explains when precision weights are justified. The public page is a preview; the full lesson requires membership. That distinction helps prevent interpreting a robust residual weight as an independently known measurement precision.

Sources

MASS, Robust Fitting of Linear Models, Arguments, Details and Value: weight-ratio convention, scale choices, local minima and convergence reporting.
MASS rlm.R source, rlm.default iteration and psi.huber: residual reweighting, weighted regression and the Huber weight function.
SAS/STAT 9.2, The ROBUSTREG Procedure: M Estimation, M Estimation, Algorithm and Convergence Criteria: fixed versus estimated scale, leverage limitations and numerical stopping checks. This is historical software documentation, used for its algorithmic discussion.
R stats, Fitting Generalized Linear Models, method, Details and Value: IWLS fitting and the distinction between prior and working weights.
R stats glm.R source, glm.fit working-response and weight updates: inverse-link derivative, variance and weighted least squares.

Turning a robust score into a weighted fit

A weighted mean that needs another iteration

What convergence does—and does not—establish

The other common IRLS: GLM fitting

Sources

Related terms

Generalised linear model

Ordinary least squares

Outlier

Residual