Word2Vec is a family of shallow models that learns a fixed vector for each vocabulary word from its neighbouring words in a text corpus. It can turn filing language into numerical features for a financial-text classifier. A nearby vector means similar usage under the training design; it does not guarantee the same accounting meaning, sentiment or expected return.
The learned vectors are a form of text embedding. Training labels come from the text itself, making this self-supervised learning. The central choice is which direction to predict, followed by how to train that prediction efficiently.
Two prediction directions
For the illustrative fragment “credit spreads widened”, take “spreads” as the centre word and a window of one word on each side.
| Architecture | Input | Training target |
|---|---|---|
| Continuous bag of words (CBOW) | Aggregate the vectors for “credit” and “widened” | Predict “spreads” |
| Skip-gram | The vector for “spreads” | Predict “credit” and “widened” as separate surrounding-word targets |
CBOW pools its context without preserving the order of those input words. Skip-gram constructs word–neighbour pairs. Here context means nearby corpus positions, not an economic regime or the full information set of an investor. A symmetric window can include words on either side; the task is not necessarily next-token prediction. These are the two architectures in Mikolov et al., Efficient Estimation, §§3.1–3.2.
Prediction direction and output objective are different choices
In the usual full-softmax or negative-sampling formulation there are two embedding tables: an input vector and an output/context vector for each vocabulary type. A pair score is their dot product . The two roles are trainable parameters, not two context-dependent senses of a word.
| Output objective | What is trained? | Vocabulary probabilities? |
|---|---|---|
| Full softmax | Compete against every output word through a common denominator | Yes: normalized over the vocabulary |
| Hierarchical softmax | Predict binary branches along a tree path to an output word | Yes: leaf probabilities sum to one; output parameters belong to internal tree nodes |
| Negative sampling | Distinguish observed pairs from sampled noise pairs | Binary discrimination scores; no normalized word distribution is supplied by this loss |
For full softmax, the conditional probability of a context word is
where is the vocabulary. Hierarchical softmax changes the output parameterization; it does not use the same per-word output table. Negative sampling changes the objective, rather than simply estimating this denominator with a handful of words. See Mikolov et al., Distributed Representations, §§2–2.2.
Read a negative-sampling loss correctly
Let be one observed context for centre word , and let be draws from a specified noise distribution. One realized skip-gram negative-sampling loss, using natural logarithms, is
The first term rewards a high observed-pair score. Each subsequent term rewards a low noise-pair score. A “negative” is a sampled training label, not a claim that the pair is linguistically impossible. The original work samples using a distribution proportional to unigram frequency raised to the power . Both that distribution and the number of draws affect the training problem. Original negative-sampling objective, §2.2.
A fixed-score toy. Suppose one observed pair has dot product and one sampled noise pair has dot product . The observed pair gets . For the noise pair, the probability assigned to its correct label is ; its observed-pair score would be .
Thus the summed, rather than averaged, loss is
Original fixed-score example. The bars show two separate binary classification terms, not a probability allocation across vocabulary words.
The two values of 0.75 have different correct labels. Adding them is not a test that word probabilities sum to one. More generally, three candidate pairs with zero scores each have sigmoid value 0.5, summing to 1.5. A sigmoid applied separately to each word score does not create a next-token probability law.
Which vectors should be exported is a separate decision. The original implementation, in
TrainModel, writes its input table syn0 when saving word vectors. Its negative-sampling output table is syn1neg. Using output vectors or combining tables changes the representation; record the convention before comparing neighbours.Static geometry, financial evaluation
Once training is frozen, a word type has the same input vector in every occurrence. Its training context shaped that vector, but a new sentence does not change the lookup. BERT instead produces contextual token representations from surrounding text. Google's historical BERT explanation makes this distinction explicitly. BERT still begins with token embeddings; its contextual output is the relevant comparison.
For filings, record the corpus cutoff, tokenisation, phrase handling, window, dimension, frequency filtering, objective and exported table. General news and financial disclosures need not put a term in the same neighbourhood. Two opposite sentiment words can share contexts, and a mean of word vectors does not automatically preserve negation or word order.
A nearest-neighbour list is a diagnostic of the fitted geometry. For a disclosure classifier, evaluate against held-out labels on the intended documents. For an investment feature, separately evaluate feasible decision-time availability and later performance. Training even an unlabelled embedding on future filings can introduce look-ahead bias; the absence of return labels does not make future text available in the past.
The Embeddings lesson develops corpus choices, document aggregation and task-specific validation. Text Representation develops the measurement choices that precede an embedding. Both have public previews; full lessons require membership.
Sources
- Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, Efficient Estimation of Word Representations in Vector Space, 2013, §§3.1–3.2: CBOW and skip-gram architectures.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean, Distributed Representations of Words and Phrases and their Compositionality, 2013, §§2–2.3: full softmax, hierarchical softmax, negative sampling and subsampling. Historical method paper.
- Tomas Mikolov's word2vec implementation,
InitNet,TrainModelThreadandTrainModel: separate parameter arrays, training branches and saved input vectors. Read as the original implementation, not a claim about defaults in every current library. - Google Research, BERT README, “What is BERT?”: context-free versus contextual representations. Historical project documentation; the repository is archived.