Sequence-to-sequence model: Definition & Examples

A sequence-to-sequence model, or seq2seq model, maps an ordered input sequence to an ordered output sequence, whose length may differ. In quantitative research, this provides a way to turn a financial headline into a sequence of structured tags or a historical observation window into a forecast path. The sequence mapping specifies the task; it does not establish predictive skill or prevent look-ahead bias.

Encode the source, decode the target

In the familiar encoder–decoder design, an encoder represents the source sequence and a decoder produces the target. The original Sutskever, Vinyals and Le model used separate long short-term memory networks: one compressed the input into a fixed-dimensional vector, and the other generated the output conditioned on that vector.

An autoregressive decoder predicts each token using the source and the preceding target tokens. For a target of length

M

, its conditional sequence probability is

p_\theta(y_{1:M}\mid x)=\prod_{t=1}^{M}p_\theta(y_t\mid x,y_{<t}).

Here

\theta

denotes learned parameters and

y_{<t}

means the target prefix before position

t

. In a variable-length token model,

M

includes an end-of-sequence token, EOS: the probability of stopping belongs in the sequence probability too.

Seq2seq does not require autoregressive decoding. Gu et al. developed a non-autoregressive translation model that produces output tokens in parallel. Nor does seq2seq require a recurrent network: the name describes a sequence-to-sequence mapping, rather than a particular neural-network layer.

The target must be shifted

Teacher forcing trains an autoregressive decoder using the true past target tokens. Suppose the target is A, B. The decoder inputs are START, A, while the targets scored at those positions are A, B. Feeding A, B as both decoder inputs and same-position targets would reveal the answer being predicted.

At inference, those true past outputs are unavailable; the decoder instead consumes its generated prefix. If it generates a wrong first token, later predictions may encounter a history unlike those seen in training. This mismatch is often called exposure bias. Bengio et al. studied it through scheduled sampling, which mixes true and model-generated previous tokens during training. Attention alone does not remove the mismatch.

Fictional fixed-length seq2seq example: decoder inputs START and true A score targets A and B with probabilities 0.8 and 0.6, giving joint probability 0.48, summed NLL 0.733969 nats and mean NLL 0.366985 nats per token. Inference feeds generated previous tokens instead.

Original fixed-length toy. Teacher forcing shifts the true target right by one position; inference replaces the true previous token with a generated one. The illustration uses a fixed context, without attention.

For the figure, condition explicitly on a fixed output length of two tokens; no EOS probability is being modelled. Given a particular source $x$ , suppose the model assigns

p(A\mid x,L=2)=0.8,\qquad p(B\mid x,A,L=2)=0.6.

The probability of the observed target A, B, conditional on that source and length, is

0.8\times0.6=0.48

. Its summed negative log-likelihood is

-\ln(0.48)=-\ln(0.8)-\ln(0.6)\approx0.733969\text{ nats}.

Averaging over the two scored tokens gives

0.366985

nats per token. These are different loss reductions of the same example, so report which one is used. The second probability is evaluated after true A; it need not remain

0.6

if inference generates another first token. A teacher-forced likelihood is therefore not a test of how well a complete generated sequence behaves.

Attention changes the source connection

A fixed context must carry everything the decoder needs. Encoder–decoder attention instead lets each output step form a weighted read of the encoder's source representations. Bahdanau, Cho and Bengio introduced a learned alignment mechanism that gives different target positions different context vectors. Attention is an optional extension to seq2seq, rather than part of its definition.

The SelfMadeQuant lesson Attention as Query–Key–Value Weighting and an Encoder–Decoder Bridge develops this weighted-context mechanism. Its preview is public; the full lesson requires membership.

For a forecasting application, all encoder inputs must be available at the forecast origin. During evaluation, generate the future path under the inputs available in use; supplying realised future values as decoder inputs measures teacher-forced prediction instead. Evaluate the resulting path for the intended task as well as its training loss. The architecture itself supplies neither a trading rule nor evidence of out-of-sample profitability.

Sources

Sutskever, Vinyals and Le (2014), Sequence to Sequence Learning with Neural Networks, §2: fixed-context encoder–decoder, conditional factorisation and EOS.
Bahdanau, Cho and Bengio (ICLR 2015), Neural Machine Translation by Jointly Learning to Align and Translate, §§2.1–3.2: encoder–decoder background and learned source alignment.
Bengio et al. (2015), Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks, §§2.1–2.4: true versus generated decoder histories and the training/inference mismatch.
Gu et al. (ICLR 2018), Non-Autoregressive Neural Machine Translation, §§2.1–2.2: autoregressive training and parallel output generation.

Encode the source, decode the target

The target must be shifted

Attention changes the source connection

Sources

Related terms

Autoregressive generative model

Conditional probability

Look-ahead bias