Gradient clipping: Definition & Examples

Gradient clipping limits the size of a gradient before an optimiser uses it to update model parameters. For a neural network forecasting volatility, an unusually large training gradient can otherwise produce a disruptive update. Clipping supplies a guardrail for that training step; it does not establish that the forecast is useful out of sample.

The usual global norm version shrinks the whole gradient by one common factor when its length exceeds a threshold. Pascanu, Mikolov and Bengio (2013), Section 3.2, explain this strategy for exploding gradients in recurrent networks. Clipping does not restore a gradient that has vanished.

One length, one scaling factor

Let $g$ be the finite gradient vector for the current update, and let $c>0$ be the chosen maximum Euclidean norm. Ideal global L2 clipping is

\widetilde g= \begin{cases} g, & \lVert g\rVert_2\le c,\\ \displaystyle\frac{c}{\lVert g\rVert_2}g, & \lVert g\rVert_2>c. \end{cases}

A zero gradient stays zero. An oversized nonzero gradient keeps its direction and ends at length

c

. The threshold is a hyperparameter, not an estimate of market volatility or a bound on portfolio risk.

Global means across the selected parameters. Flatten their gradient tensors conceptually into one vector, compute one norm, then apply one multiplier to every selected gradient entry. Clipping each layer separately is a different rule. The current PyTorch source documentation describes this aggregate convention for clip_grad_norm_.

Take the synthetic two-coordinate gradient $g=(3,4)$ and threshold $c=2$ . Its length is $\sqrt{3^2+4^2}=5$ , so the multiplier is $2/5=0.4$ :

\widetilde g=(1.2,1.6),\qquad \lVert\widetilde g\rVert_2=\sqrt{1.2^2+1.6^2}=2.

Both coordinates retain the original ratio. This remains the calculation if the entries 3 and 4 belong to separate parameter tensors included in the same global operation.

Clipping values changes the question

Coordinate value clipping truncates each gradient entry to a prescribed interval. With bounds

-2

and

2

, the same example becomes

(2,2)

. Its length is

\sqrt8\approx2.83

, which exceeds 2, and its direction changes. Bounding every coordinate does not impose the same Euclidean norm bound. PyTorch exposes this separate operation as clip_grad_value_ in its source documentation.

A synthetic gradient (3, 4) has length 5. Global L2 clipping at 2 rescales it to (1.2, 1.6), on the radius-2 circle along the same direction. Coordinate clipping at plus or minus 2 yields (2, 2), a square corner of length square root of 8, approximately 2.83, in a different direction.

Original synthetic geometry in arbitrary gradient-coordinate units. The circle bounds the L2 norm; the square bounds individual coordinates. Arrows show gradients, while a descent update points the other way.

The figure also explains why the choice matters beyond a threshold's numerical value: a circle and a square permit different gradients. Neither operation clips model weights, input observations or forecast returns.

A gradient bound is not every optimiser's step bound

For plain gradient descent with scalar learning rate $\eta>0$ and no additional update terms,

\Delta\theta=-\eta\widetilde g \quad\Longrightarrow\quad \lVert\Delta\theta\rVert_2\le\eta c.

That conclusion uses the particular update rule. Momentum carries information from previous gradients; Adam also rescales coordinates using moment estimates. Clipping today's gradient alone does not impose this same bound on their actual parameter displacement. Extra terms such as decoupled weight decay also need separate accounting.

The SGD, Momentum, RMSProp, and Adam lesson shows those update rules side by side. Its preview is public; the full lesson requires membership.

Put clipping at the intended point

Compute the gradient first, then clip before the optimiser step. When accumulating microbatches for one effective batch, clipping their final aggregate is different from clipping each microbatch contribution before adding it.

With mixed-precision loss scaling, unscale first. PyTorch's Automatic Mixed Precision examples, under “Working with Unscaled Gradients,” place scaler.unscale_(optimizer) after accumulation and before clipping. Otherwise, the nominal threshold acts on artificially scaled gradients.

The ideal formula assumes finite values. Clipping is not a repair for NaNs or infinities: the current PyTorch clip_grad_norm_ interface offers error_if_nonfinite=True to raise on a nonfinite total norm; its default is false. Its implementation also adds a small denominator stabiliser, so the worked example describes ideal arithmetic rather than exact framework output. Check the implementation and its documented options for the version used.

Record unclipped norms and how often clipping activates. Frequent clipping can motivate checking loss scaling, feature scaling and the learning rate. Making a training run numerically stable does not remove look-ahead bias or validate an economic signal.

Sources

Pascanu, Mikolov and Bengio (2013), On the difficulty of training recurrent neural networks, Sections 2 and 3.2, Algorithm 1: exploding gradients and norm rescaling.
PyTorch, Gradient clipping source and docstrings, clip_grad_norm_, _get_total_norm, _clip_grads_with_norm_ and clip_grad_value_: aggregate scope, value clipping, denominator stabiliser and nonfinite option. Accessed 1 November 2026.
PyTorch, Automatic Mixed Precision examples, “Working with Unscaled Gradients,” “Gradient clipping” and “Gradient accumulation”: unscaling and effective-batch order. Accessed 1 November 2026.

One length, one scaling factor

Clipping values changes the question

A gradient bound is not every optimiser's step bound

Put clipping at the intended point

Sources

Related terms

Adam optimiser

Hyperparameter

Look-ahead bias

Mixed precision training

Momentum and Nesterov acceleration

Vanishing and exploding gradients