Text tokenisation: Definition & Examples

Text tokenisation segments text into units called tokens; a model tokenizer also maps those units to integer vocabulary IDs. A token may be a word, part of a word, punctuation or a representation of bytes. In financial text work, this choice determines which features a filing produces and how much of a disclosure fits into a model input.

Tokenisation precedes a text embedding: the ID identifies a vocabulary entry, while the embedding supplies its numerical representation. ID 12 does not mean twice as much as ID 6. The numbers are labels, not financial quantities or semantic distances.

Two words, three tokens

Take the exact string interest rates. For this illustration, give the tokenizer the following fictional vocabulary and segmentation. Preserve the space, perform no normalisation and add no special tokens.

Piece, with quotes delimiting its characters	Vocabulary ID	Character count
`"interest"`	7	8
`" rate"` — starts with a space	12	5
`"s"`	3	1

The pieces concatenate to the original text and produce the ID sequence [7, 12, 3]. There are 14 characters including one space, two words under whitespace splitting, and three tokens under this fictional rule. All characters here are ASCII, so the UTF-8 encoding also occupies 14 bytes. That equality is incidental: the single Unicode code point € occupies three UTF-8 bytes. A user-perceived character can itself contain several code points, so “character count” also needs a convention outside this ASCII example.

Fictional segmentation of interest rates: 14 ASCII characters and two whitespace-separated words become three pieces, interest, space rate and s, with IDs 7, 12 and 3. The space is part of the second piece.

Given fictional vocabulary, not a tokenizer trained on data. The label [space] marks the single blank; it is not an extra input character. No special tokens are added.

The vocabulary was supplied, not learned by BPE or taken from a commercial model. Nothing here establishes a universal characters-per-token ratio. Another tokenizer can split the same phrase differently and assign entirely different IDs. Even a vocabulary alone may leave several possible segmentations: the segmentation rule selects among them.

How subword vocabularies are learned

Whole-word vocabularies need a policy for unseen names and spellings. Subword methods can represent many unfamiliar words by reusing smaller pieces. Those pieces need not coincide with meaningful roots or suffixes.

Byte-pair encoding (BPE) repeatedly merges frequent adjacent symbol pairs in its training corpus. In the character-based adaptation of Sennrich, Haddow and Birch (2016), words begin as characters plus an end marker; each step adds the most frequent eligible pair as a new symbol. At encoding time, the learned merge operations determine the split. Despite the name, that paper merges character sequences; byte-level BPE variants begin from a byte representation. Boundaries and preprocessing are part of the particular tokenizer.

WordPiece is another way to build a subword vocabulary. The GNMT paper describes vocabulary construction driven by language-model likelihood. This differs from selecting BPE's most frequent pair. Do not treat “WordPiece” as a synonym for BPE or assume their training objectives and encoding rules coincide.

Unigram tokenisation starts with candidate pieces and estimates their probabilities. Kudo's method prunes candidates according to their contribution to corpus likelihood. It scores a segmentation using the product of its piece probabilities; it can select the best segmentation or sample alternatives for training. It does not reconstruct a BPE merge history. Here “unigram” describes the segmentation model, not a claim that the downstream language model ignores word order.

For a pretrained model, use its matching tokenizer and vocabulary. Substituting another tokenizer changes what the IDs denote. Training a tokenizer on financial language may alter sequence lengths, but it does not by itself teach a model accounting or establish predictive value.

The space and the decimal point matter

The Tokenizers pipeline documentation separates normalisation, pre-tokenisation, the learned segmentation model and post-processing. Lowercasing, Unicode normalisation and punctuation splitting can change what reaches the vocabulary. Post-processing may add special tokens. The printed pieces are also a display convention: SentencePiece, for example, represents whitespace with ▁. A marker shown by one library is not a universal token format.

In a filing pipeline, inspect strings such as BRK.B, US10Y, 25bps, 2.5% and USD 1.2bn before choosing cleaning rules. A period may belong to a ticker or a decimal; a percent sign or unit suffix carries economic meaning. Splitting a unit across pieces need not destroy it when the model retains the sequence. Deleting punctuation or dropping the suffix before tokenisation can destroy it. These are diagnostic examples, not claimed token counts for a particular model.

A bag-of-words representation may intentionally use word-like tokens and discard order. A Transformer model instead consumes an ordered token sequence. The appropriate preprocessing depends on that downstream representation; a stop-word recipe for a count matrix should not be applied automatically to a pretrained language model.

Count the actual model input

Context capacity is measured using the model's tokenisation and input format. Instructions, document text, conversation formatting and special tokens can all occupy space; generation also needs room under the system's applicable limits. Count the final formatted input rather than estimating capacity from pages or English words. Truncation can otherwise remove the covenant qualification or risk-factor sentence the analysis needed.

Under a simple tariff with separate rates per million input and output tokens, the token charge is

C=\frac{n_{\mathrm{in}}r_{\mathrm{in}}+n_{\mathrm{out}}r_{\mathrm{out}}}{10^6}.

Here the counts are billable tokens and the rates are currency per million tokens. This is an illustrative billing structure, not a product quotation; it excludes any separately priced categories or charges. No context size, price or model-specific count is asserted here.

For reproducible research, record the tokenizer version or artifact, vocabulary, normalisation, special-token policy and truncation rule alongside the model checkpoint. Preserve the original text so a changed cleaning rule can be audited. If fitting a tokenizer for a historical forecasting test, fit it within the eligible training corpus rather than using future disclosures to choose its vocabulary.

Text Representation develops the corpus and feature choices behind this step. Transformers connects token occurrences to contextual representations and financial measurement checks. Both offer public previews; the full lessons require membership.

Sources

Sennrich, Haddow and Birch (2016), Neural Machine Translation of Rare Words with Subword Units, §3.2 and Algorithm 1: character-based BPE training and application of learned merges.
Wu et al. (2016), Google’s Neural Machine Translation System, §4.1: likelihood-based WordPiece vocabulary construction.
Kudo (2018), Subword Regularization, §3.2: unigram probabilities, best segmentation and vocabulary pruning.
Hugging Face, The tokenization pipeline, normalisation through post-processing; documentation checked 1 November 2026.
Google, SentencePiece: Whitespace as a Basic Symbol, whitespace representation; documentation checked 1 November 2026.

Two words, three tokens

How subword vocabularies are learned

The space and the decimal point matter

Count the actual model input

Sources

Related terms

Bag of words and TF-IDF