BERT (Bidirectional Encoder Representations from Transformers) is a pretrained bidirectional text encoder: it learns contextual representations, including by predicting selected tokens from a corrupted input. In financial research, that representation can support a classifier for disclosure passages. Recovering words during pretraining does not itself teach the model a financial label or establish a return forecast.
The original BERT uses a Transformer encoder. Its attention mechanism can use text on both sides of a position. “Bidirectional” describes context within the supplied sequence; it does not mean observing future market events. Google's original BERT explanation contrasts contextual representations with a single fixed vector for each word type.
Selection is different from replacement
In the historical recipe in Devlin et al., §3.1, 15% of WordPiece token positions are selected for prediction. Conditional on a position being selected, its input is replaced with
[MASK] 80% of the time, a random token 10% of the time, and left unchanged 10% of the time. The target remains its original token in every branch. WordPiece is a text-tokenisation method; its units need not be whole words.Consider a stipulated allocation across 2,000 eligible positions, ignoring special tokens, per-sequence rounding and prediction caps. Select 300 positions, then assign 240 to the mask branch, 30 to random replacement and 30 to the unchanged branch. These counts illustrate the proportions; they are not a deterministic training-batch result. Conditional on 300 selected positions, actual random branch counts vary.
Original schematic allocation matching the historical BERT proportions. The replacement percentages use the selected group as their denominator; actual stochastic allocations vary.
An unchanged selected position is still a prediction target. A random replacement can also happen to equal the original token, so “random replacement” does not guarantee a visibly changed input. The released data generator, function
create_masked_lm_predictions, preserves original labels independently of replacement. It also excludes [CLS] and [SEP] from candidates and applies rounding and a prediction cap; the illustrative pool above is not a reproduction of its exact batch construction.A compact expression for the masked-language loss is
where is the nonempty set of selected positions, is the original token and is the corrupted sequence. The sum includes selected unchanged positions; unselected positions supply context without being MLM targets in this expression. The targets come from the text itself, making this self-supervised learning.
Two pretraining tasks, then a different task
Original BERT also used next sentence prediction (NSP). In the paper's construction, half the text pairs use a genuinely following span and half a randomly sampled span. The model classifies the pair; it does not generate the next sentence. Here “sentence” can mean a contiguous text span. This is a historical BERT objective, not a requirement imposed on every later masked language model. Devlin et al., §§3–3.2.
Downstream fine-tuning starts from pretrained weights and trains for a labelled task. For example, a researcher could define “contains a covenant breach” versus “does not contain a covenant breach”, add a classification head and update the encoder and head on labelled passages. This proposed use is distinct from continuing MLM pretraining on unlabelled filings. Neither procedure supplies an audited codebook by itself. Google's overview of the two stages explains the transfer from pretraining to task-specific learning. Current Transformers documentation likewise distinguishes masked-LM and sequence-classification model heads.
An autoregressive language model instead predicts the next token using preceding tokens. Original BERT's bidirectional MLM is a different objective. A token-recovery probability is neither the probability that a passage is negative nor the probability that an asset will fall.
The financial clock still applies
Suppose a full earnings-call transcript becomes available after the call ends. BERT may use both earlier and later words when classifying that completed transcript at an eligible later decision time. Using the same full transcript to reconstruct a decision made halfway through the call would introduce unavailable text. Bidirectional attention neither creates nor cures that look-ahead bias.
Audit the checkpoint's training coverage and the fine-tuning labels as well as document timestamps. A later checkpoint applied to an old filing is not automatically a model that could have been used then. Keep the tokenizer, checkpoint, text versions, label rules and evaluation split reproducible. Test classification against held-out human labels on the intended financial medium; test any investment rule separately with feasible timing and an untouched evaluation period.
The Transformers lesson develops contextual representations and their validation deliverable. NLP Signal Validation develops the document-availability and research-selection gates. Both offer public previews; full lessons require membership.
Sources
- Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019, §§3–3.2: encoder, input spans, original MLM/NSP objectives and fine-tuning.
- Google Research, original BERT README, “What is BERT?”: contextual representations and pretraining versus downstream use. Historical project documentation; the repository is archived.
- Google Research, released pretraining data generator,
create_masked_lm_predictionsandwrite_instance_to_example_files: replacement branches, original labels, excluded special tokens, rounding and caps. The README notes this is not the exact code used for the paper. - Hugging Face, Transformers BERT documentation, model overview,
BertForMaskedLMandBertForSequenceClassification; checked 1 November 2026. No installation, model-size or benchmark claim is assumed here.