A trainable table maps category identities to vectors. Understand lookup arithmetic, one-hot equivalence, unknown categories and point-in-time identity.
Categorical embedding
A categorical embedding represents each value of a categorical feature by a vector of learned numbers. In a neural model of trading costs, for example, a venue identifier can select a short vector that the prediction network combines with order size and market conditions. The identifier supplies identity; its numerical code supplies no economic ordering.
This is one form of categorical feature encoding. It is useful when a feature has many distinct values and the downstream task can benefit from a compact, trainable representation. It does not ensure that rare venues acquire reliable estimates or that embeddings outperform a simpler encoding.
The table behind the representation
For one categorical field with K possible values and an embedding dimension d, an ordinary trainable embedding table has Kd scalar parameters. A vocabulary maps category names to integer row indices; the embedding layer retrieves the corresponding row. The integer is an address, not an input magnitude. The current Keras Embedding documentation describes this mapping from integer indices to fixed-size dense vectors.
Consider three fictional venues, with these stipulated, dimensionless vectors:
Category
Lookup ID
Embedding vector
Venue A
0
(1,0)
Venue B
1
(0,1)
Venue C
2
(1,1)
The table contains 3×2=6 parameters. Venue B selects (0,1). If a linear prediction head has weights (2,−1) and no bias, its output is
2×0−1×1=−1.
This is an illustrative score, not a return, a probability or a fitted execution-cost forecast. The head's weights are additional model parameters, excluded from the six table entries. A separate unknown-category row would also add entries; this tiny table has none.
Fictional venue B has lookup ID 1 and selects vector (0,1) from three rows with two coordinates each. The table has 6 parameters. A no-bias head with weights (2,−1) gives score −1.
Row lookup selects a vector; the prediction head turns that vector into a score. IDs carry no ordinal distance.
Relabel the venues and permute the table rows consistently, and the model returns the same vectors. Feeding codes 0, 1 and 2 directly into a numerical model instead could impose a spurious ordering or spacing.
What learning changes—and what it does not
In supervised entity embeddings, the table is trained jointly with the prediction network by minimising its task loss. Guo and Berkhahn's entity-embedding paper, Section V, makes the useful connection explicit: lookup is algebraically equivalent to multiplying a one-hot row vector by the embedding matrix, without a bias term. The implementation can retrieve a row without materialising the one-hot vector.
Consequently, an embedding table and a one-hot input followed by a bias-free linear layer of the same width have the same number of weights. Compactness comes from the chosen intermediate width and architecture, not a universal parameter saving over equivalent one-hot processing. Shared downstream layers can use patterns across categories, but every category still has its own row; an infrequent category may remain poorly estimated.
Target encoding is a different construction. It typically replaces a category with a regularised target statistic, such as a smoothed category mean. An embedding instead learns vector coordinates through the model's loss; its coordinates need not equal target means. The official scikit-learn TargetEncoder documentation explains its target statistics and cross-fitting protection. Both methods can use labels, so both need an honest training boundary.
Nor is embedding distance automatically economic similarity. The objective may make nearby vectors useful to the fitted task, but coordinates are not uniquely identified. In the toy example, double every first coordinate and halve the head's first weight: all three scores stay unchanged, while some Euclidean distances change. Architecture and regularisation can constrain such transformations; the prediction loss alone does not supply a unique economic geometry. A cluster is neither a causal relationship nor evidence that two venues have interchangeable liquidity.
Keep identity and learning inside the information boundary
Freeze the vocabulary and table together for evaluation. Fit any data-derived vocabulary, rare-category grouping and embedding weights on the training portion of each split; use validation data to choose dimension and regularisation, then assess the held-out period without refitting. Fitting a representation on future outcomes before a historical backtest creates look-ahead bias. This follows the training-only transformation principle in scikit-learn's data-leakage guidance.
An unseen category cannot select a well-trained new row by itself. Define a fallback: an explicitly trained unknown bucket, grouping with a supported category, or a separate model based on observable attributes. The Keras StringLookup documentation distinguishes vocabulary adaptation from unknown-category handling. Reserving a row does not give it useful training evidence.
For financial identities, preserve the mapping known at the prediction date. A reused ticker, changed venue code or retrospective issuer merge must not silently rewrite the past. Keep the effective and availability dates of mapping changes in a point-in-time dataset. If the intended use is prediction for entirely new issuers, test held-out identities as well as later dates: strong results on familiar identifiers need not transfer to unseen ones.
