A company name, a sentence about a covenant breach and a summary of management's outlook are different outputs. NLP covers all of them; a word-count sentiment score is only one possible application. Rules, statistical classifiers and neural language models can each perform parts of the work.
Start with the output you need
| Task | Possible financial use | What needs checking |
|---|---|---|
| Entity and event extraction | Identify an issuer, the parties to an acquisition, or a stated payment date | Correct entity, role, date and source span; an announcement is not a completed transaction |
| Classification | Label a disclosure as discussing litigation, liquidity or another defined subject | Agreement with a written codebook on held-out examples |
| Retrieval | Find passages relevant to a research question across a filing archive | Whether relevant evidence is retrieved and belongs to the right issuer and period |
| Summarisation and generation | Produce a concise briefing from selected disclosures | Each factual claim against its source, including qualifications and uncertainty |
Named-entity recognition finds mentions such as organisation names; linking a mention to a particular security or issuer is an additional problem. Text embeddings encode text as vectors that can support retrieval or classification. They are representations, not sentiment labels by definition. Retrieval-augmented generation supplies retrieved material to a generator; retrieving a relevant passage still leaves its use in the answer to be checked.
A count you can reproduce
Freeze these conventions:
- Lowercase the text and take each contiguous run of ASCII letters
athroughzas a token. - Keep every token. Apply no stemming, lemmatisation or stop-word removal.
- Declare the complete toy positive-word lexicon to be
{strong, growth}. Count matched token occurrences, then divide by all token occurrences.
company, reports, strong, growth. The match indicators are 0, 0, 1, 1: two matches across four tokens, so the positive-word hit fraction is 2/4 = 0.5. This proportion is neither the probability that the headline is positive nor a predicted return.
For this example, the arithmetic is the easy part. A real tokenizer must make choices about company names, decimals, hyphens, contractions and languages without spaces. A subword tokenizer used by a language model can split the same headline differently. Its token count need not equal the word count above. An empty token sequence has no defined hit fraction under this convention; record it as missing rather than quietly assigning zero.
Counting words is a measurement choice
A lexicon is transparent, but its matches need interpretation. The phrase “not strong” still contains a hit under the toy rule: it has no negation handling. “Growth” could describe costs, a competitor or a past period. The entity, what is growing and the surrounding sentence matter. Discarding those relationships can erase precisely the information a researcher wanted.
Language quality and investment value have different tests
A useful extraction system can save research time without predicting prices. A well-measured tone feature can describe disclosures without producing an implementable trading edge. To investigate a trading application, preserve point-in-time data: the original document version, its publication time, receipt time and the processing delay before the output could be used. A fiscal period-end is not the release time, and an edited transcript cannot silently replace the version available earlier.
The model has a clock too. A later pretrained checkpoint may contain subsequent filings or event outcomes. Calling a historical prompt “zero-shot” does not establish that its response was obtainable then. Document the checkpoint, permitted knowledge, prompts and any fitted preprocessing; distinguish retrospective analysis from a feasible historical strategy test.
Finally, check rights for the actual corpus, dictionary and model use. A readable news page or transcript does not itself establish permission for bulk collection, redistribution or commercial processing. Reproducibility includes documenting those inputs and permitted uses, alongside the transformation that produced the feature.
Sources
- Daniel Jurafsky and James H. Martin, Speech and Language Processing, third-edition online manuscript, released 19 August 2026. The chapter map spans tokens, classification, embeddings, retrieval, generation, named entities and information extraction.
- Stanford NLP Group, CoreNLP overview, “About,” “Pipeline” and “Annotations”: examples of linguistic processing beyond keyword counts.
- Tim Loughran and Bill McDonald, When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks, Journal of Finance 66(1), 2011, pp. 35–36, on financial context and dictionary misclassification; official dictionary repository for list versions and stated use terms. The headline and lexicon above are original teaching examples.