Stochastic depth gates residual branches during training. Compare original and inverted scaling, enumerate a two-branch chain, and see why mean matching is not exact ensemble prediction.
Stochastic depth
Stochastic depth randomly bypasses whole residual branches during training while retaining the shortcut path. It regularises a residual network by exposing its shared parameters to networks of varying effective depth. For a financial forecasting encoder, this changes the training procedure; its value must still be established on eligible later data.
Here qℓ is the survival probability, not the drop probability. When bℓ=0, the branch contributes nothing and the shortcut passes the input onward. The gate belongs on the residual branch, before addition; masking the sum would also erase the shortcut.
This equation isolates the addition without a subsequent activation. The original paper includes a post-addition ReLU; its identity bypass relies on nonnegative incoming activations. Shape-changing blocks also need a compatible shortcut rather than an unqualified identity.
The paper uses unscaled surviving branches in training and multiplies each branch by its survival probability at inference:
xℓ+1test=xℓtest+qℓFℓ(xℓtest).
Thus all branches are available at inference. “Randomly shallower during training” does not mean permanently deploying a smaller model.
Enumerate a two-branch network
Consider an original synthetic scalar chain starting at 10, with constant residual contributions 2 and 4. Masks are independent, with survival probabilities q1=0.8 and q2=0.5. All quantities are dimensionless; there is no fitted market model.
First mask b1
Second mask b2
Probability
Training output 10+2b1+4b2
0
0
0.10
10
0
1
0.10
14
1
0
0.40
12
1
1
0.40
16
The complete probability-weighted mean is
0.1(10)+0.1(14)+0.4(12)+0.4(16)=13.6.
Scaled inference also gives 10+0.8(2)+0.5(4)=13.6. The expected number of active residual branches during training is 0.8+0.5=1.3, compared with both branches at inference. This counts branches, not individual layers or elapsed computation time.
Four possible training outputs have probabilities: 10 at 10%, 12 at 40%, 14 at 10%, and 16 at 40%. Their mean is 13.6. Both residual branches are active at inference, scaled by survival, giving 13.6.
Original exact enumeration of a synthetic constant-branch chain. The dashed line is the probability-weighted training mean; the inference value matches it in this restricted example.
Matching this mean is not exact ensemble prediction for a nonlinear network. Append the readout g(y)=max(0,y−14). Only the outcome 16 contributes, so the mean readout is 0.4(2)=0.8. Deterministic inference instead gives g(13.6)=0. Input-dependent residual branches introduce further nonlinear composition. Replacing each mask by its mean cannot generally reproduce averaging all network outputs.
Check the scaling and the mask shape
Current Torchvision source uses inverted survival scaling: for qℓ>0, the training branch is (bℓ/qℓ)Fℓ(x) and inference leaves Fℓ(x) unchanged. Its argument p is the drop probability, 1−qℓ. Do not apply the original inference multiplier on top of this convention.
With the same fixed constants, inverted scaling gives a training mean and ordinary inference value of 16, rather than 13.6. This compares two operators on stipulated branches; it is not a claim that models trained under the two conventions learn identical parameters.
Torchvision's batch mode shares one gate across the whole mini-batch. Its row mode samples a gate per example, broadcasting it across that example's remaining dimensions. Neither mode independently drops each feature or token. Its function receives an already computed branch tensor, so zeroing that tensor alone does not skip the preceding branch computation. Actual speed savings require an execution path that avoids that work.
Huang, Sun, Liu, Sedra and Weinberger (2016), Deep Networks with Stochastic Depth, Section 3, Equations 2–5: branch gating, survival schedules, effective depth and original inference scaling. The enumeration and nonlinear readout here are original.
