Lion optimiser: Definition & Examples

The Lion optimiser is a neural-network training method that moves each parameter according to the sign of a mixture of its current gradient and stored momentum. For a quant researcher fitting a forecasting or document model, its attraction is a smaller optimiser state than Adam, with a different trade-off between gradient magnitude and direction. It changes how parameters are fitted; predictive value still has to be established on data outside training.

Lion was introduced by Chen and colleagues in Symbolic Discovery of Optimization Algorithms. The name abbreviates Evolved Sign Momentum. Its defining detail is that the mixture used to move today is different from the momentum saved for tomorrow.

Two mixtures, one stored memory

Let $g_t$ be the loss gradient at the current parameter vector $\theta_{t-1}$ , and initialise $m_0=0$ . In the original algorithm,

\begin{aligned} c_t &= \beta_1m_{t-1}+(1-\beta_1)g_t, \\ \theta_t &= (1-\eta_t\lambda)\theta_{t-1} -\eta_t\operatorname{sign}(c_t), \\ m_t &= \beta_2m_{t-1}+(1-\beta_2)g_t. \end{aligned}

All operations act coordinate by coordinate. Here $\eta_t$ is the learning rate, $\lambda$ is decoupled weight decay, and $\beta_1,\beta_2\in[0,1)$ are averaging coefficients. The paper's standard choice is $\beta_1=0.9$ , $\beta_2=0.99$ .

With those coefficients, today's mixture puts 10% weight on the current gradient; the saved memory puts 1% on it. Both use the previous momentum. The author implementation forms the sign update before overwriting that memory.

The sign is

+1

-1

or zero. Excluding decay, every nonzero coordinate moves by exactly

\eta_t

. A tiny positive mixture therefore produces the same positive-direction sign as a large positive one. This is different from gradient clipping, which caps magnitude while retaining information about smaller gradients. Nor is Lion Adam with its denominator removed: Adam's saved moments and update rule differ.

A reversal that the saved memory does not show

Take one fictional scalar parameter with $\theta_0=1$ , $m_0=0$ , constant $\eta=0.1$ , zero decay and the coefficients above. Feed in gradients $g_1=2$ and $g_2=-1$ . These are prescribed inputs to illustrate the update, not measured model performance.

Step	Incoming gradient	Direction mixture $c_t$	New parameter $\theta_t$	Saved momentum $m_t$
1	$2$	$0.2$	$0.9$	$0.02$
2	$-1$	$-0.082$	$1.0$	$0.0098$

At step 2,

c_2=0.9(0.02)+0.1(-1)=-0.082, \qquad m_2=0.99(0.02)+0.01(-1)=0.0098.

The current mixture is negative, so the parameter increases by $0.1$ . Yet the saved momentum is positive. Using $\operatorname{sign}(m_2)$ instead would move the parameter the wrong way for Lion, producing $0.8$ rather than $1.0$ .

Fictional scalar Lion updates with zero weight decay: step 1 has gradient 2, direction mixture 0.2, parameter 0.9 and saved momentum 0.02. Step 2 has gradient minus 1, direction mixture minus 0.082, parameter 1 and saved momentum 0.0098. The second direction mixture and saved momentum have opposite signs.

Lion uses c for the current direction and m for future memory. On step 2 their signs disagree; the parameter follows c.

Returning to the starting parameter does not mean training has reset: its memory is now $0.0098$ , not zero.

Compare training runs, not default settings

Lion stores one momentum array; ordinary AdamW stores two moment arrays. At equal state precision that halves those accumulator bytes, not total training memory: model parameters, gradients, activations and other buffers remain. The authors' README describes this saving and the need to retune learning rate and decay when switching from AdamW. Copying an AdamW learning rate is not a controlled comparison.

The decay factor above depends on

\eta_t\lambda

. Changing the learning rate while holding

\lambda

fixed also changes per-step shrinkage. Moreover, sign steps need not become small merely because gradients become small; a suitable learning-rate schedule still matters.

The original paper's limitations include tasks where Lion performs similarly to AdamW. Its benchmarks do not establish superiority on financial time series. For a financial experiment, keep the chronological split and information available at each date fixed, tune on validation data, and compare predictive error as well as training cost. That is an experimental implication, not a result reported for a trading strategy in the Lion paper.

Reading Loss Curves develops the logging and validation discipline needed for that comparison. Its preview is public; the full lesson requires membership.

Sources

Chen, X. et al. (2023), Symbolic Discovery of Optimization Algorithms, Sections 3.1–3.2, 5–6 and Appendix A, Algorithm 2.
Google Research, Lion PyTorch implementation, initialisation and step; checked 1 November 2026. The author repository is archived and read only.
Google Research, Lion README, memory accounting and hyperparameter guidance; checked 1 November 2026.

Two mixtures, one stored memory

A reversal that the saved memory does not show

Compare training runs, not default settings

Sources

Related terms

Adam optimiser

Gradient clipping

Learning-rate schedule