Dropout randomly masks activations during training. Follow inverted scaling, see why a preserved activation mean is not an exact ensemble prediction, and distinguish MC dropout uncertainty.
Dropout
Dropout is a regularisation method that randomly sets selected neural-network activations to zero during training. It discourages reliance on particular combinations of units, aiming to reduce overfitting. For a financial forecasting network, dropout changes how the model learns; it does not establish that a fitted signal will survive later market conditions.
A masked unit is temporarily absent from that forward pass, not permanently deleted. Different masks train different subnetworks that share parameters. This is the model-combination motivation in Srivastava and coauthors' original paper, rather than a collection of independently fitted models.
First settle what the probability means
Here p is the drop probability, and q=1−p is the keep probability. This agrees with the current PyTorch Dropout documentation in its source. The original paper instead uses p for retention: copying a number without its convention can reverse the intended mask rate.
For a fixed scalar activation h, inverted dropout uses a Bernoulli keep mask m:
m∼Bernoulli(q),h=qmh,0<q≤1.
Kept activations are amplified during training; ordinary inference switches the dropout operation to identity, returning h. Section 10 of the original paper describes this alternative to scaling weights at inference. Do not apply both corrections.
Conditioning on the activation before masking gives
E[h∣h]=h,Var(h∣h)=qph2.
This expectation concerns the mask randomness at fixed h. Other layers and parameters can also change during training.
One activation, the entire distribution
Take the stipulated, dimensionless activation h=8, with drop probability p=0.25 and keep probability q=0.75. There are exactly two outcomes:
Mask outcome
Probability
Training activation
Dropped
1/4
0
Kept
3/4
8/(3/4)=32/3
Their probability-weighted mean is 8. Their variance is
41(0−8)2+43(32/3−8)2=364.
Usual inference returns 8, without this mask variability. Matching the mean does not make the training and inference distributions identical.
The full training activation distribution has mass 25% at zero and 75% at 32/3, approximately 10.67. Its mean is 8 and variance 64/3. Usual inference returns 8. After max of zero and activation minus 8, the mask-averaged output is 2 but the dropout-off output is zero.
Original synthetic example with dimensionless activations. The bars exhaust the probability distribution; they are not a simulation. Preserving the activation mean does not preserve a downstream nonlinear prediction.
Now put a nonlinear operation after dropout: g(z)=max(0,z−8). The dropped branch yields 0 and the kept branch yields 8/3. Averaging those outputs gives 2, whereas switching dropout off gives g(8)=0. Thus E[g(h)]=g(E[h]) in this example. An exact activation-mean identity is not an exact whole-network ensemble identity.
Inference with masks is a separate choice
Monte Carlo dropout keeps dropout active for repeated predictions with different masks, then summarises their outputs. Gal and Ghahramani (2016), Sections 3–4, give an approximate Bayesian interpretation through a restricted variational distribution over masked weights. That interpretation connects the training objective to a prior, likelihood and regularisation settings; random masks alone do not make an arbitrary network an exact Bayesian neural network.
Spread among stochastic network outputs describes variation in the fitted function under the mask distribution. It is not automatically the full uncertainty of a future observation. In the paper's Gaussian regression construction, predictive variance also includes an observation-noise term. Increasing the number of passes reduces Monte Carlo error in summaries of the chosen approximation; it does not fix the approximation or guarantee calibrated intervals or better point forecasts.
When sampling masks, keep batch-normalisation layers in their intended inference mode; their stored statistics and behaviour are separate from the dropout choice.
For a return forecast, assess those claims on later eligible observations, including changed market conditions. Dropout cannot repair look-ahead bias, and mask disagreement is not a direct estimate of trading risk.
The mask's shape matters too
For sequences, specify whether the mask is redrawn at each time step or held fixed across a sequence. Gal and Ghahramani's recurrent-network variant repeats a mask through time, including on recurrent connections. This sequence-locked construction differs from independently masking every time step. A dropout rate alone does not specify that choice.
Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Sections 1, 4 and 10: shared subnetworks, Bernoulli masks and training-time scaling. The two-point example and nonlinear counterexample here are original.
PyTorch, Dropout source documentation, Dropout class: drop-probability convention, training scaling and evaluation identity. Accessed 1 November 2026.
