Vanishing and exploding gradients: Definition & Examples

Vanishing and exploding gradients are failures of scale in a neural network’s backward pass: derivatives propagated through many operations become too small to convey a useful learning signal, or so large that they can destabilise training. In a financial sequence model, this can prevent a distant observation from influencing learning even when the architecture can represent that dependence.

A small gradient is not automatically a fault: near a stationary point, it may be appropriate. The concern here is attenuation or amplification along the computational path between the loss and earlier computations.

Twenty multiplications, very different outcomes

Take a deliberately linear scalar chain with twenty steps, each multiplying its input by a fixed number $a$ . Let the loss be the final output itself, so its derivative with respect to that output is one. The chain rule gives

\frac{dL}{dx_0}=\prod_{k=1}^{20}a=a^{20}.

With $a=0.5$ , the derivative is $2^{-20}=0.00000095367431640625$ . With $a=1.5$ , it is approximately $3{,}325.25673008$ . Every local derivative is finite and nonzero; multiplication alone creates the extreme scale difference. With $a=1$ , the derivative remains one.

Log-scale chart of synthetic scalar gradient chains starting at 1. After 20 derivative factors, 0.5 per step gives 0.00000095367431640625, 1 per step gives 1, and 1.5 per step gives 3325.256730079651.

Original scalar-chain illustration. The vertical axis is logarithmic: equal vertical distances represent equal multiplicative changes. Each chain uses a constant local derivative and a unit incoming gradient.

These are synthetic sensitivities, not trained forecasting results. Real networks have changing derivatives, multiple paths and parameter-gradient contributions that can add or cancel. The toy isolates a mechanism, rather than claiming that depth always causes it.

The vector version depends on direction

For a layer $x_k=f_k(x_{k-1})$ , let $J_k$ be its Jacobian matrix and use column gradients $g_k=\nabla_{x_k}L$ . The backward recursion is

g_{k-1}=J_k^{\mathsf T}g_k.

Repeated matrix multiplication can contract some directions while expanding others. In a recurrent neural network, unrolling through time produces the same kind of chain. The final parameter gradient also sums contributions from different times; losing a long-range contribution need not make the total gradient zero. Pascanu, Mikolov and Bengio, §§1.1–2.1, develop this distinction.

Do not diagnose the effect from individual weight magnitudes alone. Activation derivatives also enter the Jacobians, and matrix directions matter. Saturated sigmoid or tanh units can have very small derivatives; a ReLU on its inactive side has zero derivative. ReLU therefore removes one saturation mechanism without guaranteeing healthy gradient flow throughout a network. Glorot and Bengio examine activation and gradient scales and derive an initialisation scheme under explicit approximations.

Match the intervention to the failure

Initialisation and activation choice: aim to start with usable forward and backward scales. The appropriate variance depends on the activation and layer widths; an initialisation argument does not guarantee stability throughout training.
Residual connections: a compatible block $y=x+F(x)$ has Jacobian $I+J_F$ . This supplies an identity contribution, as analysed by He et al., §2. It does not guarantee a nonzero total derivative: the scalar choice $F(x)=-x+3$ makes the block constant and its derivative zero. See residual network for shortcut and activation-order details.
Gradient clipping: bounds the supplied gradient according to a chosen rule, helping contain large excursions. It cannot recover a contribution already erased by the derivative chain. Clipping also does not by itself bound every adaptive optimiser’s complete parameter update; see gradient clipping.

Inspect gradient norms by layer or temporal distance, activation distributions, and actual parameter-update sizes. A single overall norm can hide weak early-layer signals. Loss spikes warrant checking finite inputs, loss scaling and learning rate as well as gradient propagation; a flat loss alone does not identify vanishing gradients.

SelfMadeQuant’s Backpropagation as the Chain Rule develops the backward-pass calculation. Its preview is public; the full lesson requires membership. For financial models, solving an optimisation problem still does not establish predictive value: evaluate the resulting model on appropriately later observations, keeping preprocessing and model selection inside their eligible data windows.

Check the mechanism

A scalar path has successive local derivatives $2$ , $1/4$ and $2$ , with incoming loss derivative one. Must its gradient explode because two factors exceed one?

Answer. No. The product is

2\times(1/4)\times2=1

. The full product matters; counting expanding steps is insufficient. For vector paths, matrix order and directions matter too.

Sources

Pascanu, Mikolov and Bengio (2013), On the difficulty of training recurrent neural networks, §§1.1–2.1 and 3.2. Temporal contributions, Jacobian products and norm clipping.
Glorot and Bengio (2010), Understanding the difficulty of training deep feedforward neural networks, §§3–4. Activation saturation and initialisation analysis.
He, Zhang, Ren and Sun (2016), Identity Mappings in Deep Residual Networks, §§2 and 4. Identity paths and activation ordering.

Twenty multiplications, very different outcomes

The vector version depends on direction

Match the intervention to the failure

Check the mechanism

Sources

Related terms

Gradient clipping

Neural network

Neural weight initialisation

Residual network