A neural architecture with learned gates that control how much of a representation is transformed and how much is carried through a layer.
Highway network
A highway network is a neural network whose layers use learned gates to control the flow between a transformed representation and a carry path. For a quant building a deep feature encoder, this creates a way to retain useful features across layers while learning where to change them.
Two paths, one learned gate
The general highway layer has a transform H, a transform gate T and a carry gate C:
y=T(x)⊙H(x)+C(x)⊙x.
Here ⊙ means elementwise multiplication. The widely used coupled carry version sets C(x)=1−T(x). That coupling is a design choice, not a requirement of every highway architecture. Shapes must match at the merge. The original paper, §2, gives both forms.
A usual gate is an input-dependent sigmoid,
T(x)=σ(WTx+bT),σ(z)=1+e−z1.
Each component can therefore carry or transform a different amount. With coupled gates, a gate near zero mostly carries the input; a gate near one mostly uses H. A finite sigmoid approaches these endpoints without attaining them. These weights are not probabilities that one model is correct, nor percentages of neurons switched off.
Follow one coordinate
Take a fictional, dimensionless hidden coordinate x=2. Suppose its transformed value is H(x)=5 and the transform gate returns T(x)=0.20. In the coupled version,
y=0.20×5+0.80×2=1.0+1.6=2.6.
Synthetic scalar highway layer: input 2 is transformed to 5, weighted by transform gate 0.20 to contribute 1.0; the carry branch weights input 2 by 0.80 to contribute 1.6. Their sum is 2.6.
A coupled carry gate blends two representations of the same input. The gate value is stipulated for this example, not estimated from market data.
An independent way to read the calculation is
y=x+T(x)(H(x)−x)=2+0.20(5−2)=2.6.
The gate moves this coordinate one fifth of the way from its existing value to its transformed value. This is a coordinatewise interpolation for this coupled sigmoid design. It does not require a single shared gate across all features.
What the carry path does—and does not—provide
A residual network typically adds a learned correction to an identity shortcut, y=x+F(x). The highway formula above instead gates the transform and carry branches. Although it can be rewritten as a gated correction to x, simply adding x+H(x) would be a different computation. He et al., §§2–3.2, explicitly distinguish their ungated identity shortcuts from highway gates.
A carry path also does not guarantee that gradients survive. For a differentiable scalar coupled layer, the total derivative is
dxdy=T(x)H′(x)+1−T(x)+(H(x)−x)T′(x).
The last term matters because the gate itself depends on the input. A nonzero carry contribution can be offset by the other terms; for example, a constant half-open gate and H(x)=−x produce an identically zero output. Gate saturation and products of layer derivatives still matter for vanishing and exploding gradients.
The original authors used negative transform-gate biases to favour carrying early in training and reported easier optimisation in their tested deep networks. This was an initialisation strategy and empirical result, not a guarantee for arbitrary architectures or financial datasets. See Training Very Deep Networks, §§2.2–3.1.
In a financial encoder, carrying a feature across depth means retaining its representation within the current input's computation. It does not automatically create memory across market timestamps. The numerical example demonstrates branch accounting only; predictive value still has to be established on data unavailable during fitting.
