A large language model learns patterns in tokens and context. Separate its next-token objective from factual accuracy, saved memory and point-in-time financial evidence.
Large language model
A large language model (LLM) is a large neural model trained on extensive language data to learn statistical patterns in tokens and their context. A token is a unit of encoded text, which can be a word, part of a word or punctuation. In quant research, an LLM can help extract a covenant from a filing, label earnings-call passages, summarise evidence or draft research code. Whether the result is accurate, reproducible or useful for a trading decision needs a separate test.
“Large” has no universal parameter threshold. Most contemporary generative LLMs use transformers, a neural-network architecture that makes token representations depend on context. The broader language-model family also includes models trained to recover masked text. This article focuses on the causal, next-token formulation behind many text generators; Hugging Face's documentation distinguishes that objective from masked language modelling.
What the token objective actually rewards
For tokens x1,…,xT, an autoregressive model factorises a sequence probability as
pθ(x1,…,xT)=t=1∏Tpθ(xt∣x<t),
where θ denotes learned parameters and x<t denotes preceding tokens. In practice the available history is bounded by the model's context mechanism. A common pretraining loss averages negative log-probabilities of the observed next tokens. Text supplies the targets, so this is self-supervised learning: a human need not label every next token.
Consider a synthetic vocabulary containing only Token A, Token B and Token C, after one fixed context. The model assigns probabilities 0.5, 0.3 and 0.2, summing to one. If the observed next token is Token C, its negative log-likelihood is
ℓ=−lnpθ(Token C∣context)=−ln(0.2)≈1.60944 nats.
For this single one-hot target, that is also the cross-entropy loss. Natural logarithms give units of nats. The training objective rewards assigning more probability to the token that actually follows in the corpus.
Synthetic next-token probabilities are 0.5 for Token A, 0.3 for Token B and 0.2 for the observed Token C. The negative log-likelihood of Token C is 1.60944 nats; no truth probability is measured.
Original synthetic three-token example at a fixed context. The observed token determines the loss; the probabilities describe text continuation, not factual correctness or a market outcome.
There is no event called “the answer is true” in this probability calculation. A high-probability continuation can repeat an error in the corpus or supply a plausible but unsupported figure. Even a model's generated phrase “90% confident” is an output to evaluate, not automatically a calibrated probability. To interpret a score as a probability of correctness, define that event and test calibration against observed answers on the relevant task.
Modern assistants often undergo post-training, including instruction tuning and preference-based training; some also learn from verifiable task rewards. These change behaviour beyond base pretraining. They do not turn each token probability into a truth probability. Jurafsky and Martin's Post-training chapter, introduction and §8.1, explains why next-token pretraining alone does not supply instruction-following or reliable problem-solving.
Context, learned parameters and saved memory
These are three different places information can reside:
Location
What it contributes
What it does not establish
Learned parameters
Patterns and information acquired during pretraining or later training
A complete, current database with recoverable provenance for every answer
Current context
Instructions, examples, document excerpts and other inputs available for this generation
Permanent learning of the supplied material
External storage
Saved records, conversation summaries or a searchable document collection that an application can supply later
That the generator has received, used or correctly interpreted every stored record
Giving examples in a prompt can change the answer without changing model weights. Brown et al.'s few-shot study, §2, explicitly separates this in-context conditioning from fine-tuning. A conversation saved by an application is therefore different from a model being trained on it.
A context window bounds how many tokens the model can process under its supported setup. Instructions, documents, conversation history and generated output can compete for a finite budget; exact accounting depends on the system. A long filing archive can exceed it. Splitting, retrieving or summarising documents changes the evidence presented and can lose a qualification, table heading or cross-reference. Fitting a document into the window also does not prove that the model used every relevant passage correctly. Record what actually reached the model, including any truncation or summaries.
Retrieval-augmented generation supplies selected external passages at generation time. It can make newer or private material available without retraining the base model. The original RAG paper, §§1–2, distinguishes parameter-based information from an external document index. Retrieval still has its own failure modes: the index may omit a document, search may miss it, or a passage may concern the wrong issuer or reporting period. The generator can then misread a correct passage. A citation must be opened and checked against the particular claim; its presence is not proof of support. Jurafsky and Martin's RAG discussion, §11.4, also treats irrelevant and incorrect retrieved passages as an unresolved input problem.
The model has an information clock too
Suppose a researcher scores a filing released in 2019 to reconstruct a decision in 2020, using a checkpoint trained later. The filing may be historically eligible while the model contains knowledge of subsequent events. An instruction to “use only what was known in 2020” does not remove information from learned parameters.
Fictional timeline: a 2019 original filing precedes a 2020 decision, but a 2025 model checkpoint may encode an event outcome from 2023. Audit document availability and model or retrieval knowledge separately.
Original fictional timeline. Correctly dating the input filing does not by itself make a later checkpoint a point-in-time research instrument.
This is a potential source of look-ahead bias. Glasserman and Lin's financial-news study, introduction and §2, examines overlap between model training and backtest periods, alongside interference from general company knowledge. Its experiments are historical evidence about a particular model and design, not a current model specification or a promise of profitable implementation. Anonymising issuer names can be a diagnostic; distinctive products or events can still reveal identity.
For a historical study, preserve the checkpoint or model version, what is known about its training coverage, prompts, decoding settings, actual responses and run dates. A published training-data cutoff is a coverage statement, not a guarantee that every earlier fact was learned or a complete audit of contamination. If training provenance is insufficient, describe the work as retrospective and state the unresolved information boundary. A genuinely prospective assessment after the system is frozen provides stronger evidence of historical feasibility.
Apply the same point-in-time dataset discipline to retrieval: original versus amended filing, publication and receipt time, index version and processing delay. Today's search results cannot silently serve as yesterday's information set. Also distinguish a model that was available at the decision date from a later model used to study old documents.
Test the research job, then its economic use
An extraction task needs checks of the entity, period, units, exact source span and missing values. A summary needs claim-by-claim support and preservation of material qualifications. A label needs a fixed codebook and comparison with audited human labels on held-out documents. Keep near-duplicate filings and repeated boilerplate from making the held-out sample an accidental repeat of training or prompt-development data.
The LLM Annotation lesson develops the codebook and human–model comparison. The NLP Signal Validation lesson adds document availability, research selection and later confirmation. Both have public previews; full teaching requires membership.
Tool access adds another dependency. A calculator can evaluate an expression correctly while the model supplies the wrong cash flow, currency or annualisation convention. Search can return a current document while the question concerns an earlier version. Generated code needs execution against checked cases. Evaluate the complete process, retain tool inputs and outputs, and specify when insufficient evidence should lead to an abstention or review.
Finally, useful research assistance and a tradable signal have different targets. After validating the text output, any investment claim still needs an untouched forward evaluation of a frozen decision rule with feasible timing, costs and execution. Fluency, low token loss and strong generic benchmarks cannot substitute for that evidence.
Sources
Hugging Face, Causal language modeling, opening definition and distinction from masked language modelling; current documentation checked 29 October 2026. No software installation or model-specific limit is assumed here.
Tom B. Brown et al., Language Models are Few-Shot Learners, 2020, §2 and §4: in-context conditioning versus parameter updates, and benchmark contamination.
