A squeeze-and-excitation (SE) block is a neural-network unit that summarises each feature channel, learns gates from the joint summary, and rescales the channels. In a quantitative forecasting model, it offers a way to adapt the use of learned features to the current input window; it does not establish that those features predict returns.
Context first, then rescaling
In a convolutional neural network, a channel is a feature map produced by a learned filter. SE adds three operations. For channels and positions per channel:
Squeeze averages positions within each channel. Excitation jointly processes those means through a narrower fully connected layer, ReLU, a layer restoring channel count, and an elementwise sigmoid. The learned weights are fitted with the enclosing network. Rescaling broadcasts each gate across its channel without changing the tensor shape. Hu, Shen and Sun, §§3.1–3.2, define this construction.
Two channels, two positions
Take synthetic, unitless feature maps. Channel 1 contains ; channel 2 contains . Their squeeze means are and . Suppose the excitation network returns gates and . These gates are stipulated for illustration: no particular learned weights or training outcome is being asserted.
| Channel | Input positions | Squeeze mean | Gate | Rescaled positions |
|---|---|---|---|---|
| 1 | 1, 3 | 2 | 0.25 | 0.25, 0.75 |
| 2 | 2, 6 | 4 | 0.80 | 1.60, 4.80 |
The first channel retains one quarter of each activation; the second retains four fifths. Their gate sum is . Sigmoid gating does not impose the sum-to-one constraint of softmax, so these values are neither portfolio weights nor a probability distribution over channels. “Excitation” here means relative emphasis: a sigmoid gate between zero and one does not increase an activation's absolute magnitude.
Original illustrative calculation: the joint mean descriptor informs channel gates; each gate multiplies every position of its channel. The gates are stipulated, not outputs of specified fitted weights.
What the gates can tell you
SE is often called channel attention. It supplies one scale per channel, rather than pairwise attention between positions. In the original SE-residual network, the gates rescale the residual branch before its addition to the shortcut (§3.3).
The squeeze descriptor discards positional order: reversing each example channel leaves its mean unchanged. The rescaled feature maps still retain that order. Thus pooling the descriptor does not reduce the whole output to two numbers.
A large gate is not a causal importance score. It acts on an internal learned representation, whose scale and downstream connections also affect the prediction. Channel 2's gate of does not mean that a corresponding economic variable explains 80% of the outcome.
Pool only what the forecast can know
For a time-series adaptation, global pooling must respect the prediction cutoff. A single prediction at today's close may pool a historical window ending today. If the network instead emits a prediction at every timestamp, pooling the entire sequence would let an early prediction depend on later observations. Even causal convolutions cannot repair that look-ahead bias; the pooling domain must also be restricted appropriately.
The worked calculation demonstrates rescaling, not forecasting skill. Evaluate the complete model against a suitable baseline on later data, including the choice to add SE within the training and model-selection process.
For the surrounding shape and skip-connection mechanics, SelfMadeQuant's CNN Building Blocks, Receptive Fields, and Residual Design Patterns has a public preview; the full lesson requires membership.
Sources
- Hu, Shen and Sun, Squeeze-and-Excitation Networks, 2018, §§3.1–3.3, equations (2)–(4): spatial average pooling, bottleneck excitation, sigmoid channel rescaling and SE-ResNet placement. The original application is computer vision; the timing discussion above concerns an adaptation to financial forecasting.