Batch normalisation: Definition & Examples

Batch normalisation is a neural-network transformation that, during training, centres and rescales each feature using statistics from a mini-batch, then applies a learned scale and offset. It can make a neural network easier to optimise. For a quant researcher, the crucial extra consequence is that one observation's training output depends on the other observations grouped with it.

A feature is compared with its neighbours

Consider a dense layer whose activation table has examples in rows and features in columns. Batch normalisation calculates a separate mean and variance down each column, across the batch. It does not pool unrelated features into one statistic. For one feature with batch values

x_1,\ldots,x_m

\mu_{\mathcal B}=\frac{1}{m}\sum_{i=1}^{m}x_i, \qquad v_{\mathcal B}=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_{\mathcal B})^2,

\widehat{x}_i=\frac{x_i-\mu_{\mathcal B}}{\sqrt{v_{\mathcal B}+\epsilon}}, \qquad y_i=\gamma\widehat{x}_i+\beta.

Here the variance uses divisor

m

, not

m-1

. The positive constant

\epsilon

stabilises the denominator;

\gamma

and

\beta

are learned per feature alongside the network's other parameters. These are the operations in Ioffe and Szegedy's Algorithm 1. In convolutional networks, the normalisation group also includes spatial locations within each channel; the dense-table description does not specify every possible tensor layout.

Contrast this with layer normalisation: on this two-dimensional table, it computes statistics across the features within each row. Changing another example can change a training-mode batch-normalised value; it does not change that row's layer-normalised value. The Keras layer-normalisation documentation makes this distinction explicit.

Two values, then a learned transformation

Take a fictional feature with batch values

1

and

3

. Its mean is

2

and its divide-by-two variance is

1

. For this nonzero-variance arithmetic example only, set

\epsilon=0

. Normalisation gives

-1

and

1

. With

\gamma=2

and

\beta=1

, the final outputs are

-1

and

3

The final outputs therefore have mean $1$ and variance $4$ . Calling the entire layer's output “zero mean and unit variance” misses the learned affine transformation. Even before that transformation, positive $\epsilon$ makes the batch variance of $\widehat{x}$ equal to $v_{\mathcal B}/(v_{\mathcal B}+\epsilon)$ , rather than exactly one.

Synthetic scalar batch values 1 and 3 have mean 2 and variance 1. With epsilon zero for this toy, normalised values are minus 1 and plus 1; gamma 2 and beta 1 give minus 1 and plus 3. With positive epsilon 0.00001, activation 3 normalises to approximately plus 0.999995 in batch 1,3 but to zero in batch 1,3,5.

Original synthetic calculation. The upper panel isolates the arithmetic; the lower panel uses positive epsilon and holds one activation fixed while changing its batch neighbours. All values are dimensionless.

A zero-variance group is different from division by zero: with positive epsilon, centring makes every value zero and the final output is $\beta$ . A library may reject a training input with only one value per feature, but the formula itself does not produce an infinite normalised value in that case.

The model has statistical state

In common implementations, inference substitutes stored training estimates for the current batch's statistics. A prediction can then be evaluated without depending on which other observations happen to arrive alongside it. Those stored means and variances belong in the saved model checkpoint, together with the learned parameters.

The exact convention is library dependent. Keras BatchNormalization documents batch statistics for training and moving statistics for inference. PyTorch's BatchNorm1d documentation and source distinguish the divide-by-

m

training-forward variance from the unbiased variance used to update its running estimate. With track_running_stats=False, PyTorch also uses batch statistics during evaluation. “Evaluation mode always uses running statistics” is therefore too broad.

These choices can change outputs even when the learned weights are identical. Record the normalisation configuration and framework version when reproducing a result; do not reconstruct inference by copying the last training batch's mean and variance.

A batching choice can become a timing assumption

Suppose a model scores securities at different times during a trading day. If its normalisation statistics combine an early observation with later observations, the early score depends on information unavailable at its intended decision time. No future target label is needed for this problem: the activations alone supply the extra information. This is a form of preprocessing leakage when the evaluation claims to reproduce those earlier decisions.

A contemporaneous cross-section can be legitimate when every constituent input is available before the joint decision. The research contract must say whether predictions are made jointly or sequentially. Temporal or cross-sectional batching is not automatically safe or automatically invalid.

For an ordinary held-out evaluation, keep the validation period out of updates to running statistics and use the inference behaviour intended for deployment. Leaving BN in training mode can both make predictions depend on validation-batch composition and update the stored state with validation observations. Disabling gradient calculation alone does not select inference behaviour. Any deliberate online adaptation needs a separately specified, time-consistent evaluation.

Small or highly correlated batches can give noisy or unrepresentative statistics; increasing their size does not create independent market observations. Stored statistics can also become unrepresentative after a regime change. There is no universal best batch size, and easier optimisation does not establish better out-of-sample trading performance.

What the mechanism does—and what it explains

The original paper motivated BN through reducing internal covariate shift, meaning changes in layer-input distributions during training. That motivation is not a proven universal explanation of its success. Santurkar and colleagues challenged the link in their experiments and studied smoother optimisation behaviour as an alternative explanation. Their results do not turn BN into a guarantee of stable gradients or successful generalisation in every architecture.

For the surrounding training loop, Mini-Batches, Vectorization, Autograd, and a Reproducible MLP Workflow connects batch shapes with chronological splits, frozen preprocessing and checkpoints. The public page is a preview; the full lesson requires membership.

A feature is compared with its neighbours

Two values, then a learned transformation

The model has statistical state

A batching choice can become a timing assumption

What the mechanism does—and what it explains

Sources

Related terms

Layer normalisation

Neural network