Adam optimiser: Definition & Examples

The Adam optimiser is a gradient descent method that updates each model parameter using running averages of its gradients and squared gradients. Its name comes from adaptive moment estimation. In a quant model, this changes the path taken while fitting a loss function: two weights can receive different effective step sizes even when they share one learning rate. It does not establish whether the fitted signal survives a chronological test or trading costs.

Two memories, one update

For a gradient $g_t$ evaluated at the current parameters, Adam keeps a signed average $m_t$ and a squared-gradient average $v_t$ . Starting with both at zero, the original algorithm is

\begin{aligned} m_t &= \beta_1m_{t-1}+(1-\beta_1)g_t, \\ v_t &= \beta_2v_{t-1}+(1-\beta_2)g_t^2, \\ \hat m_t &= \frac{m_t}{1-\beta_1^t},\qquad \hat v_t = \frac{v_t}{1-\beta_2^t}, \\ \theta_t &= \theta_{t-1}-\alpha_t\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}. \end{aligned}

All arithmetic is coordinatewise. The decay factors

\beta_1,\beta_2\in[0,1)

control memory;

\alpha_t>0

is the learning rate. The positive constant

\epsilon

stabilises the denominator and is outside the square root in this convention. The second accumulator estimates a raw second moment, not a variance: it does not subtract the squared mean. These are the conventions in Kingma and Ba's Algorithm 1.

A persistent gradient direction reinforces the numerator. Large recent squared gradients enlarge the denominator. Neither mechanism removes the need to choose a learning rate or learning-rate schedule.

A first step that exposes a common mistake

Consider one fictional scalar parameter, initially $\theta_0=1$ . Use $g_1=2$ , $\beta_1=0.9$ , $\beta_2=0.999$ , $\alpha_1=0.1$ and $\epsilon=10^{-8}$ . These numbers make the arithmetic visible; they are not a training recommendation.

Quantity	First-step value
Signed accumulator $m_1$	$0.2$
Squared-gradient accumulator $v_1$	$0.004$
Corrected signed accumulator $\hat m_1$	$2$
Corrected squared-gradient accumulator $\hat v_1$	$4$

The parameter becomes

\theta_1=1-0.1\frac{2}{\sqrt{4}+10^{-8}} \approx0.9000000005.

At the first step, the correction factors cancel the initial averaging weights exactly. For a nonzero scalar gradient and negligible epsilon, the update is therefore approximately one learning rate in the opposite direction to the gradient, rather than the learning rate times the raw gradient.

Fictional Adam first step: gradient 2 produces m equal to 0.2 and v equal to 0.004; correcting gives 2 and 4. The corrected step magnitude is 0.100000, versus 0.316228 when both corrections are omitted.

Both correction factors matter. This compares the same scalar first step, with epsilon outside the square root in both cases.

Omitting both corrections would give a step magnitude of about

0.316228

, over three times the corrected step. Correcting only the numerator does increase that numerator; it does not follow that correcting the entire ratio always increases the update.

Bias correction addresses the missing history caused by zero initial accumulators. Under constant first and second gradient moments, it removes that initialisation bias in the individual moment estimates. During training, those moments can change. Even unbiased moment estimates would not make their nonlinear ratio an unbiased estimate of the true gradient. This distinction follows from the moment calculation in Section 3 of the original paper.

INTERACTIVE EXPLORER

Optimiser trajectories

Compare gradient, momentum and Adam on a known convex loss.

AdamW changes where regularisation enters

With an $L_2$ penalty $\lambda\lVert\theta\rVert^2/2$ added to the loss, the gradient passed into Adam includes $\lambda\theta$ . The penalty therefore enters the moment accumulators and adaptive scaling.

AdamW instead applies weight decay separately from the loss-gradient update. One common learning-rate-scaled convention is

\theta_t=(1-\alpha_t\lambda)\theta_{t-1} -\alpha_t\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},

where the moments use the loss gradient without that penalty term. The decay shrinks parameters directly rather than passing through the adaptive denominator. Loshchilov and Hutter show why the familiar equivalence between

L_2

regularisation and weight decay for plain SGD does not generally extend to Adam. Record the decay convention and which parameters receive it when comparing experiments.

What must survive an experiment restart

Adam has a training history. Reloading model weights while resetting the moments and step counter changes subsequent updates. A training checkpoint needs the optimiser state as well as the parameters; full reproducibility also depends on the data order, random state and schedule. The PyTorch checkpoint guide explicitly distinguishes resuming training from saving weights for inference.

Implementation details also matter. In documentation checked on 1 November 2026, Keras Adam defines its epsilon as the paper's alternative “epsilon hat”, rather than Algorithm 1's epsilon. Matching parameter names and numerical epsilon values across libraries is therefore insufficient to establish identical updates. Check the actual formula, including epsilon placement and bias-correction convention.

Finally, adaptive scaling is not a convergence guarantee. The original paper's online convex convergence analysis was challenged by Reddi, Kale and Kumar, who identified a proof issue and constructed convex counterexamples, including for bias-corrected Adam. Their AMSGrad variant changes the second-moment memory and has guarantees under stated assumptions; neither result guarantees a global optimum for a neural network or profitable forecasts.

For a research workflow beyond the update rule, Reading Loss Curves explains experiment logging, validation-based selection and failed-training diagnosis. The preview is public; the full lesson requires membership.

Sources

Kingma, D. P. and Ba, J. (2015), Adam: A Method for Stochastic Optimization, Algorithm 1 and Sections 2–4. The convergence caveat above reflects subsequent work.
Loshchilov, I. and Hutter, F. (2019), Decoupled Weight Decay Regularization, Section 2 and Algorithm 2.
Reddi, S. J., Kale, S. and Kumar, S. (2018), On the Convergence of Adam and Beyond, Sections 3–4 and the bias-correction footnote in Section 2.
Keras, Adam API documentation, epsilon argument; checked 1 November 2026.
PyTorch, Saving and Loading Models, general checkpoints for resuming training; checked 1 November 2026.

Two memories, one update

A first step that exposes a common mistake

Optimiser trajectories

AdamW changes where regularisation enters

What must survive an experiment restart

Sources

Related terms

Learning-rate schedule

Loss function