Label latency is the time from a prediction event until its realised target value becomes usable by the learning system. Delayed feedback is the resulting gap between issuing predictions and being able to learn from or score their outcomes. In a trading backtest, a refit can suffer from look-ahead bias if it uses a label from a completed trade whose exit data had yet to reach the training pipeline.
What could the 10:00 fit know?
A model predicts the sign of a return measured over the next 30 minutes. In this fictional example, all times are UTC on the same day. The label is determined at the end of its return window, recorded by the source, then made available when the training pipeline can read the validated data. The fit uses a snapshot through 10:00, inclusive.
| Sample | Prediction event | Outcome determined | Source recorded | Available to training | Label in 10:00 fit |
|---|---|---|---|---|---|
| A | 09:00 | 09:30 | 11:00 | 11:00 | Exclude |
| B | 08:00 | 08:30 | 08:45 | 09:00 | Include |
Both outcomes are determined by the fit cutoff. The solid spans show the outcome windows and the dashed spans show the wait for usable labels.
The prediction horizon is 30 minutes for both rows. A's label latency is 120 minutes and B's is 60 minutes. For B, the source record existed for 15 minutes before the pipeline could use it.
If is the first time the pipeline could use sample 's label, the fit at cutoff can use that label only when . Replay features as of each original prediction event and labels as of each refit cutoff. If labels can be corrected, keep the availability time of each version and reconstruct the version the fit could actually see. A publication date alone can be too coarse to prove intraday availability. The St. Louis Fed, for example, distinguishes vintage dates and source release dates from data availability on its own services. FRED real-time periods, FRED release dates.
A pending outcome is still unknown
For a target such as “did this loan miss a payment within the observation window?”, a verified miss can establish a positive early. A negative requires the window to close and the records covering it to be complete. A blank at the cutoff could mean the window is still open or that the data confirming the outcome have yet to arrive. Treating every blank as class 0 can understate the event rate. Training on mature cohorts avoids that shortcut, but uses older examples and can miss recent shifts. For time-to-event data, a model that accounts for delay or censoring can use partial follow-up if its observation assumptions fit the data. Chapelle's conversion model is an example that separates event probability from time to the event. Chapelle, Modeling Delayed Feedback in Display Advertising.
The SelfMadeQuant lessons on Labels and Overlapping Outcomes and Observation, Event and Availability Timestamps cover fit eligibility and how to choose the clock used in a replay. Both have public previews, with full lessons for members.
Sources
- Olivier Chapelle (2014), Modeling Delayed Feedback in Display Advertising, §§1, 3 and 6.3.
- Federal Reserve Bank of St. Louis, Real-Time Periods and Release Dates.