Skip to content

Open Neural Network Exchange

ONNX packages a model as a typed computation graph for interchange. Follow a two-node export and distinguish format conformance from measured output agreement.

Open Neural Network Exchange (ONNX) is an open specification and file format for exchanging machine-learning models as computation graphs. An inference export describes the operations that produce a prediction, the parameter tensors those operations use, and the names, types and shapes of the graph's inputs and outputs. A quant team can export a model from its research framework and run it with a compatible runtime in a production risk or trading service. ONNX Intermediate Representation specification

Read the graph from input to output

Suppose a fictional scoring service receives one row with two features. All values are dimensionless float32. Its graph has two nodes, MatMul followed by Add. The caller supplies the graph input xx of shape (1,2)(1,2). The weights WW of shape (2,2)(2,2) and bias bb of shape (2,)(2,) are initializers. They are named tensors stored with the model. Here they appear only in the initializer list, so they are constants. Each node can take a stored value as an operand even though the caller supplies only xx. ONNX graph specification
The matrix multiplication uses each column of WW to produce one component of the intermediate row zz. ONNX MatMul
z=xW=[23][14−12]=[−114].z=xW=\begin{bmatrix}2 & 3\end{bmatrix} \begin{bmatrix}1 & 4\\-1 & 2\end{bmatrix} =\begin{bmatrix}-1 & 14\end{bmatrix}.
Add aligns the rank-one bias b=(0.5,−0.5)b=(0.5,-0.5) with the last dimension of zz. For broadcasting, the shape (2,)(2,) acts like (1,2)(1,2), so each bias goes to its corresponding column and the output stays a two-dimensional row. ONNX Add, multidirectional broadcasting
y=z+b=[−1+0.514−0.5]=[−0.513.5].y=z+b=\begin{bmatrix}-1+0.5 & 14-0.5\end{bmatrix} =\begin{bmatrix}-0.5 & 13.5\end{bmatrix}.
The 1 by 2 input row (2, 3) enters MatMul with stored 2 by 2 weights, yielding (minus 1, 14). Add uses the stored rank-one bias (0.5, minus 0.5) to produce the 1 by 2 output row (minus 0.5, 13.5).
The 1 by 2 input row (2, 3) enters MatMul with stored 2 by 2 weights, yielding (minus 1, 14). Add uses the stored rank-one bias (0.5, minus 0.5) to produce the 1 by 2 output row (minus 0.5, 13.5).
Synthetic two-node graph. The caller supplies x. The weight matrix W and bias b are stored as initializers and feed different nodes.
The SelfMadeQuant lesson on neural network architecture works through the same weight and bias shape logic for a batch of inputs (public preview, full lesson for members).

Three checks for an export

  1. Format conformance. The model declares an intermediate representation (IR) version for the format and imports versioned operator sets, or opsets, that determine the applicable operator definitions. The ONNX checker validates the model's consistency against the specification. ONNX versioning, ONNX checker API
  2. Runtime compatibility. Load and run the export in the target runtime build. Check its support for the model's IR version, opsets, operators and element types. In ONNX Runtime, an execution provider supplies implementations for parts of the graph that it can run on a particular backend, so confirm which provider actually executes the relevant nodes. ONNX Runtime compatibility, execution providers
  3. Measured output agreement. Run the original model and the export on the same representative feature tensors. Compare outputs with tolerances chosen for the decision that uses them, and compare decisions on inputs near any action threshold. Check for training-serving skew in any preprocessing outside the graph. A service can swap two feature columns and still supply the right tensor shape. A successful load would miss that error. ONNX Runtime model validation

In the toy graph, the hand calculation gives an expected output for one input. For a live scoring service, numerical agreement and latency both depend on what runs in the actual deployment, including its backend and input workload.

Last updated
  • model interchange
  • inference
  • model deployment
  • tensor operators
Keep connecting the ideas
← Back to the glossaryBack to top ↑