The lottery ticket hypothesis proposes that a dense, randomly initialised neural network contains sparse subnetworks whose original initial weights let them reach comparable test accuracy within at most the same number of training iterations. For quant research, this makes compression a question about trainability, as well as the number of surviving weights.
A ticket includes its starting weights
In Frankle and Carbin's original experiment, the search has three essential steps:
- Save the dense network's random initial weights.
- Train it, then use neural network pruning to select connections from the trained network, removing low-magnitude weights.
- Reset the surviving connections to their saved initial values and train this subnetwork in isolation, keeping removed connections absent.
Repeated train–prune–reset rounds form iterative magnitude pruning. A candidate becomes a winning ticket only after its retraining performance meets the stated accuracy and iteration comparison. The name describes a successful empirical comparison, not every sparse mask.
What exactly gets reset?
Take four illustrative, dimensionless weights. Their initial values and a stipulated later trained state are
Now stipulate the keep mask . A one means retain that connection; a zero means remove it. The original reset operation uses the initial values:
where means elementwise multiplication. Keeping the trained values instead gives
These are different starting points for subsequent training, despite having exactly the same surviving connections. Two of the four scalar weights remain: this ledger has 50% sparsity, counting the two zeroed positions among all four positions.
This mask is hand-chosen to isolate reset mechanics. Magnitude pruning that kept the two largest absolute trained weights would retain coordinates three and four, giving mask . It would not produce our stipulated mask. No dataset, network outputs or training runs are supplied here, so the ledger establishes neither predictive accuracy nor a winning ticket.
An original reset ledger with fixed, dimensionless values. The mask is stipulated; it neither follows from magnitude pruning here nor demonstrates a winning ticket.
Separate the experiments
| Experiment on the retained connections | Starting values | Question |
|---|---|---|
| Fine-tune a pruned model | Trained weights | Can the sparse model recover or preserve its fit? |
| Original lottery-ticket test | Saved initial weights | Could this subnetwork have learned in isolation from its original start? |
| Random-reinitialisation control | Fresh random weights, same mask | Is the selected structure sufficient without the saved initial values? |
The original paper's reinitialisation experiments found that replacing the saved initial values could substantially worsen performance. This is evidence about those settings, not a universal guarantee that fresh initialisation always fails.
Rewinding changes the starting point. Frankle et al. (2020), §4, extend the procedure by restoring surviving weights to an early training checkpoint , rather than . Their larger vision-network experiments show that this can recover matching subnetworks where resetting to initialisation fails. A checkpoint with already contains learning, so this result is distinct from discovering a trainable subnetwork at random initialisation. The authors call the rewound subnetworks matching, reserving winning ticket for the initialisation case.
What a quant experiment must keep separate
A sparse return-prediction network might match its dense baseline on one split and lose that advantage in a later market period. The lottery-ticket idea supplies an experimental question; it supplies no evidence of a financial signal or its persistence.
For a meaningful comparison, keep the data construction, optimiser, batch size, learning-rate schedule and stopping rule explicit. Include the dense baseline and the same-mask random-reinitialisation control. Discover masks from training data, select settings on chronological validation data and leave a later test period untouched. Searching masks against the final test result introduces overfitting just like searching model hyperparameters.
Also report discovery cost separately from retraining cost. Finding a mask can require training the dense network and repeating pruning rounds. A successful sparse retraining run does not erase that expenditure. An iteration count is not elapsed time, either: an irregular sparse mask may still execute through dense operations. Measure latency and memory on the intended implementation; 50% sparsity alone establishes neither a twofold speedup nor a halving of total research compute.
For background on starting-weight scale, SelfMadeQuant's initialisation lesson includes a section on Glorot and He initialisation. The preview is public; the full lesson requires membership.
Sources
- Frankle and Carbin (2019), The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, §1 for the hypothesis and reset procedure; §2 for the same-mask random-reinitialisation comparison.
- Frankle, Dziugaite, Roy and Carbin (2020), Linear Mode Connectivity and the Lottery Ticket Hypothesis, Algorithm 2 and §§4.1–4.3 for rewinding, matching subnetworks and limitations of resetting to initialisation.