Self-supervised learning builds training targets from the data itself. Trace a masked reconstruction, separate pretext loss from the downstream task, and preserve financial information timing.
Self-supervised learning
Self-supervised learning is machine learning in which the training signal is constructed from the data itself, rather than supplied as a separate annotation for the target task. A model might recover a hidden part of a record or recognise related views of the same observation. For a quant team with a large archive of filings but relatively few labelled events, this offers a way to learn representations before fitting the event predictor. It does not establish that those representations forecast returns.
“Unlabelled” does not mean “without a target.” The procedure creates a target that can be checked, then uses a loss function to train the model. Murphy's §1.3.3 describes this as creating proxy supervised tasks from unlabelled data. The proxy is often called a pretext task.
Where the target comes from
Consider a synthetic numeric record, with all values dimensionless. Save a copy of a coordinate whose value is 2, hide that coordinate from the model, and ask the model to reconstruct it from the remaining context. Suppose the reconstruction is 1. The squared loss on that hidden coordinate is
ℓpre=(x−x)2=(1−2)2=1.
The target x=2 came from the original training record. Nobody needed to annotate that record with a separate class or financial outcome. The original value enters the loss, while the model receives the masked input. This is a deliberately small accounting of the training signal; no model was fitted to produce the assumed reconstruction.
A synthetic training record contains a hidden value 2. The model receives context with a mask and reconstructs 1; the saved target 2 gives squared loss 1. The learned encoder is later adapted using separate task labels and evaluated on later data.
Original synthetic masking example. Values and squared loss are dimensionless. The target bypasses the model input and enters the loss; the reconstruction is an assumed output, not a fitted result.
Devlin et al.'s BERT paper, §3.1, uses a related idea for text: predict original tokens at selected positions from their context. Its token prediction uses a classification loss, rather than the squared numeric error in this illustration. Masking is one form of self-supervision, not a requirement for every method.
Another form is contrastive learning. Chen et al.'s SimCLR, §2.1, constructs related views of an image and learns to distinguish their relationship from other examples in the batch. The choice of transformations defines which differences the representation should tolerate. A transformation sensible for photographs may erase crucial information in market data: reversing an order-book sequence, for example, changes the temporal ordering a forecasting task depends on.
Pretext success and task success answer different questions
A representation is an encoding of the input learned while solving the pretext task. After pretraining, a researcher can reuse that encoder for a downstream task such as classifying an announcement using labelled examples. This adaptation is a form of transfer learning.
Stage
What supplies the target?
What does the result tell you?
Self-supervised pretraining
A hidden value, token or relationship derived from the training data
How well the model solves the chosen proxy objective
Supervised adaptation
The defined downstream outcome or annotation
How well the representation and task model fit that task
Held-out evaluation
Outcomes withheld from fitting and selection
Evidence about generalisation under the declared evaluation design
Downstream labels need not be manually annotated either: a future return can be computed from prices. The useful distinction here is between learning through a data-derived proxy objective and fitting the particular task the researcher wants to solve. “No manual labels” alone does not establish the usefulness of a pretraining procedure.
A linear probe freezes the encoder and trains a linear prediction head on downstream labels. SimCLR's §2.3 evaluation protocol uses this to assess its learned representations. Fine-tuning instead updates some or all of the pretrained encoder along with the task model. Compare these choices with a model trained without pretraining under the same eligible data and evaluation rules. A lower reconstruction loss need not produce a better downstream result.
For more on these adaptation choices, the representation-learning lesson develops pretext tasks and frozen probes. Its full lesson requires membership; the linked page provides a public preview.
Unlabelled history still has a clock
For a historical trading evaluation, ask what was available at each model-fitting date as well as at each prediction date. An encoder pretrained on later test-period filings has learned from future observations even if no return labels were used. Such a model does not reproduce a strategy that could have been fitted before those filings existed. A different experiment that deliberately allows access to assessment inputs must state that access; it cannot be presented as an untouched test of a previously fitted live forecasting system.
Context also has a timing boundary. A filing already released in full can be encoded using words on both sides of a masked word without borrowing a future market observation. Feeding an unreleased later filing into the representation for an earlier trade would create look-ahead bias. Bidirectional text context and future financial information are different issues.
Keep the pretraining corpus, preprocessing fits, adaptation labels and model-selection decisions within their declared information windows. Fit transformations on eligible training data and apply them unchanged to held-out data; scikit-learn's data-leakage guidance explains why preprocessing is part of fitting. For temporally dependent observations, its time-series evaluation guidance explains why future observations need a suitable chronological assessment. Self-supervision changes the source of the training signal; it does not remove those boundaries.
