TabNet learns input-specific feature masks across sequential decision steps for tabular prediction. See how masking works and why attention weights do not establish causal effects.
TabNet
TabNet is a neural-network architecture for tabular classification and regression that learns which input features to emphasize at successive decision steps. For a quant working with rows of borrower characteristics or market signals, its distinctive feature is that selection can change from one observation to another.
Selection happens inside the prediction
In Arık and Pfister's original architecture, an attentive transformer uses the preceding step's representation and a prior tracking feature use to produce a mask. Sparsemax maps its scores to nonnegative weights that sum to one and can include exact zeros. This makes feature selection part of the trained network, rather than a fixed list chosen before fitting.
The masked input enters a feature transformer: shared and step-specific layers use fully connected mappings, batch normalization and gated linear units. Its output splits into a decision component and information for the next step. Rectified decision components are summed, then mapped linearly to the output. Here, “transformer” names these blocks; it does not mean a standard query–key–value Transformer. The original paper's architecture section specifies this mechanism.
Two masks on the same row
Take a stipulated, dimensionless input row x=(2,4). Suppose its first two decision-step masks are m1=(0.8,0.2) and m2=(0.1,0.9). Elementwise multiplication, denoted by ⊙, gives
m1⊙xm2⊙x=(1.6,0.8),=(0.2,3.6).
The second step receives the original row again. Information passed between steps changes how the next mask is chosen; it does not replace the original feature values with the previous masked values.
For original dimensionless row (2, 4), step 1 mask (0.8, 0.2) gives (1.6, 0.8); step 2 mask (0.1, 0.9) gives (0.2, 3.6). Each step multiplies the original row, not the previous masked row.
Stipulated two-feature example, not a fitted TabNet model. Both steps mask the same original row; bar lengths show dimensionless mask weights, and products are the inputs to subsequent feature processing.
These masks are supplied for illustration, not estimated from data. Neither contains a zero: sparsemax permits sparse masks but does not require every possible mask to exclude a feature. The figure stops before nonlinear feature processing, so it supplies no predicted return, default probability or final feature-importance score.
Reading a mask responsibly
A weight of 0.8 describes this step's input weighting for this row. It is not a coefficient saying that one extra unit of that feature raises the prediction by 0.8. Changing an input can also change the masks and the nonlinear transformations. Comparing mask weights alone therefore cannot recover a marginal response.
Nor does attention establish causality. A credit model could select a variable that proxies for another variable. That selection would not show what changing the variable through an intervention would do to default risk. Inspect masks as evidence about the fitted model's selection behavior, alongside predictive validation and checks for unstable or redundant inputs.
Comparing it on financial data
TabNet's architecture does not establish that it will beat gradient boosting on a new dataset. Even the original paper's Appendix A reports some similar or slightly worse results against tree ensembles. Those benchmarks do not answer a particular trading or credit-model comparison.
For a forecasting application, compare candidates on the same prediction dates, information set, target and selection budget. Keep future observations out of learned preprocessing, mask fitting and tuning. The SelfMadeQuant lesson Time-Ordered Train, Validation and Test Design develops that evaluation schedule, including label-availability checks; its preview is public and the full lesson requires membership.
Sources
Arık, S. Ö. and Pfister, T., TabNet: Attentive Interpretable Tabular Learning, December 2020 preprint version, published at AAAI 2021. See “TabNet for Tabular Learning,” especially feature selection and feature processing; “Experiments”; and Appendix A. The numerical illustration here is original and stipulated.
