Model quantisation represents a model’s weights, intermediate activations, or both at lower numerical precision, for example by encoding 32-bit floating-point values as 8-bit integers. For a neural network that repeatedly scores a universe of securities, this can reduce memory use and latency. The precision change can also move scores across a ranking cutoff or an alert threshold. Neural network pruning changes which weights or structures are present. Quantisation changes the numbers the retained model can represent. Jacob et al. (2018), §§1–2
From a value to a code and back
An affine integer quantiser uses a positive scale for the spacing between reconstructed values and a zero point for the integer code representing zero. With signed int8 codes and nearest rounding, using ties to even on , the mapping is
First round and add the zero point, then clip to the allowed code range. Dequantisation converts the stored code back to its representative value . All original values mapped to the same code give the same result. This rounding and clipping convention matches the signed int8 mapping in ONNX. ONNX QuantizeLinear, ONNX DequantizeLinear
For an illustrative grid with and , the reconstruction range runs from to .
| Original value | Rounded code before clipping | Stored int8 code | Dequantised value |
|---|---|---|---|
| −0.61 | −2 | −2 | −0.50 |
| 0.13 | 1 | 1 | 0.25 |
| 0.74 | 3 | 3 | 0.75 |
| 40 | 160 | 127 | 31.75 |
The first three values incur rounding errors of at most half a grid step, . The value would land exactly on a grid point if code were available. Clipping forces it to code , leaving an error of . At fixed bit width, a larger scale covers a wider range and leaves coarser steps between nearby values.
The blue staircase maps each input to its reconstructed value. Near zero it follows the dashed exact-recovery line in small steps. The upper plateau shows how large inputs collapse to the same value.
Choosing the grid for deployment
Post-training quantisation starts with a fitted model. With static activation quantisation, a representative calibration sample supplies the activation ranges used to choose fixed parameters. If calm periods dominate that sample, check how often activations clip in volatile periods. Dynamic activation quantisation derives the parameters at inference time and adds work to the forward pass. Quantisation-aware training simulates low-precision effects during training so the model can adapt. For ONNX Runtime’s int8 quantisation tools the input model must use opset 10 or higher, and support still depends on the operators and target runtime. ONNX Runtime quantisation guide, Jacob et al. (2018), §3
An int8 weight takes one byte versus four for a float32 weight. Total model size also includes scales, zero points and any tensors left at higher precision. End-to-end latency depends on which operations use efficient kernels on the target hardware, how much time conversion takes and the batch sizes served. Compare both versions on the same inputs and serving setup, including tail latency. The SelfMadeQuant lesson on Profiling and Optimisation works through timing the real entry point and checking numerical changes with matched inputs. ONNX Runtime performance FAQ
For a ranking model, compare how often names cross the selection cutoff and whether the rank information coefficient holds up on the same held-out dates. A small average score difference can hide a change right where the portfolio selects names.