LARS and LAMB optimisers: Definition & Examples

LARS and LAMB are layer-adaptive optimisers: they rescale a training update using the size of the parameter block being changed. LARS means layer-wise adaptive rate scaling; LAMB means layer-wise adaptive moments optimizer for batch training. Both target large-batch training. For a quant training a deep forecasting model, they change how much each layer moves relative to its current weights; they do not establish that a forecast will generalise.

A trust ratio controls relative movement

Ordinary gradient descent applies a global learning rate to a gradient. LARS adds a scalar for each layer. Writing $w$ for that layer's weights and $g$ for its loss gradient, the original paper's equations 5–6 motivate

q=\eta\frac{\lVert w\rVert_2}{\lVert g\rVert_2+\lambda\lVert w\rVert_2}, \qquad w_{\mathrm{new}}=w-\alpha q(g+\lambda w).

Here

\alpha

is the global learning rate,

\eta

is the trust coefficient and

\lambda

is weight decay. This displayed update omits momentum. Decay contributes both to the denominator and to the direction being scaled; the denominator is a sum of norms, not the norm of the combined direction. The original experiments used

\eta=0.001

; it must not silently disappear into a supposedly universal LARS formula. See You, Gitman and Ginsburg, Section 4.

With zero decay, nonzero norms and no momentum,

\frac{\lVert w_{\mathrm{new}}-w\rVert_2}{\lVert w\rVert_2}=\alpha\eta.

The ratio sets the step's size relative to the layer. “Trust” here names a norm-based scale; it is not a probability that the gradient is correct.

One snapshot, two routes to the same displacement

Consider a fictional two-weight layer. To isolate the arithmetic, use trust coefficient $\eta=1$ , learning rate $\alpha=0.01$ , no momentum, no decay and no epsilon. These are teaching conventions, not default settings.

Let $w=(3,4)$ and $g=(0.3,0.4)$ . Their Euclidean norms are $5$ and $0.5$ , so the LARS-style trust ratio is $10$ . The amount subtracted is

\alpha qg=0.01\times10\times(0.3,0.4)=(0.03,0.04).

Its norm is $0.05$ , exactly $1\%$ of the weight norm. The new weights are $(2.97,3.96)$ . Plain SGD on the same gradient at the same learning rate would subtract $(0.003,0.004)$ , with norm $0.005$ : only $0.1\%$ of the weight norm.

Now stipulate that LAMB's adaptive direction is

u=(0.6,0.8)

, whose norm is

1

. With identity weight-norm scaling and zero decay, its trust ratio is

5/1=5

. The amount subtracted is again

0.01\times5\times(0.6,0.8)=(0.03,0.04)

Fictional layer with weights (3, 4), norm 5: LARS-style gradient (0.3, 0.4) has norm 0.5 and trust ratio 10; stipulated LAMB direction (0.6, 0.8) has norm 1 and trust ratio 5. Both subtract (0.03, 0.04) at learning rate 0.01, giving new weights (2.97, 3.96).

Aligned directions produce the same step in this specified snapshot. The LAMB direction is stipulated, rather than computed from the displayed raw gradient.

These directions align because

u=2g

. Multiplying a direction by a positive constant and then normalising it cancels that constant. This coincidence does not make LARS and LAMB identical. In particular, the stipulated

u

is not a calculation of Adam's first step from the displayed

g

LAMB adapts coordinates before adapting the layer

LAMB first constructs an Adam-style direction from bias-corrected first and second gradient moments. In the convention of the original paper's Algorithm 2, write

u=\frac{\hat m}{\sqrt{\hat v}+\epsilon}+\lambda w, \qquad w_{\mathrm{new}}=w-\alpha\frac{\phi(\lVert w\rVert_2)}{\lVert u\rVert_2}u.

The moment operations are coordinatewise. Decay is added after moment adaptation, but remains inside the direction whose norm sets the trust ratio. Thus it is not a separate, unscaled shrinkage step. The function

\phi

controls weight-norm scaling; the example uses

\phi(z)=z

. Coordinate adaptation can change the direction, so equal relative step lengths need not produce equal parameter changes. See You and colleagues, Section 3 and Algorithm 2.

The LAMB paper also presents a LARS formulation that normalises a momentum direction. That ordering differs from scaling gradients before accumulating momentum. Record the actual variant, decay placement, norm scaling and zero-norm handling when comparing runs. The unclipped ratios above require nonzero denominators.

Layer scaling can accompany a global learning-rate schedule. Neither its name nor large-batch benchmark results guarantees faster training or better forecasts on financial data. Compare elapsed time, update budgets and later-period validation results under the same data availability rules.

For the underlying recurrences, SGD, Momentum, RMSProp, and Adam as Optimization Choices develops gradient averaging, momentum and squared-gradient adaptation. Its preview is public; the full lesson requires membership.

Sources

You, Y., Gitman, I. and Ginsburg, B. (2017), Large Batch Training of Convolutional Networks, Section 4, equations 5–6 and Algorithm 1; Section 5 experimental trust coefficient.
You, Y. and colleagues (2020), Large Batch Optimization for Deep Learning: Training BERT in 76 minutes, Sections 3–3.2, Algorithms 1–2, and Section 4's task-specific experiments.

A trust ratio controls relative movement

One snapshot, two routes to the same displacement

LAMB adapts coordinates before adapting the layer

Sources

Related terms

Adam optimiser

Learning-rate schedule