Low-rank attention and Linformer: Definition & Examples

Low-rank attention reduces the cost of an attention mechanism by approximating its mixing of sequence positions through a smaller intermediate representation. Linformer does this by projecting keys and values along the sequence axis before attention. For a quant processing long earnings-call transcripts, the attraction is a smaller attention calculation; whether the compression preserves a short but decisive disclosure is an empirical question.

Compress positions, not features

For one head, let the already feature-projected queries and keys be $Q,K\in\mathbb{R}^{n\times d}$ and values be $V\in\mathbb{R}^{n\times d_v}$ . Ordinary scaled dot-product attention is

Y=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt d}\right)V,

with softmax applied separately to each row.

Using the orientation $E,F\in\mathbb{R}^{r\times n}$ , Linformer instead forms

\bar K=EK,\qquad \bar V=FV,\qquad \bar Y=\underbrace{\operatorname{softmax}\left(\frac{Q\bar K^\top}{\sqrt d}\right)}_{B\in\mathbb{R}^{n\times r}}\bar V.

The original paper, §4, introduces this sequence compression. The ordinary feature projections change each token's coordinates;

E

and

F

combine information across positions. All

n

query positions remain, but each scores

r

compressed slots. These slots need not correspond to individual tokens.

The authors' fairseq implementation separates feature projections from sequence-compression layers. Compression weights are trainable unless explicitly frozen. They are not generally constrained to be nonnegative or sum to one. The repository is archived; it documents the original mechanism rather than a current software recommendation.

A compression you can calculate

Take a scalar query $q=1$ , key dimension $d=1$ , keys $K=(1,2,3)^\top$ and values $V=(10,20,30)^\top$ . These are fictional numbers in arbitrary units. Choose the fixed, illustrative projection

E=F=\begin{pmatrix} 1/2&1/2&0\\ 0&0&1 \end{pmatrix}.

The first slot averages positions 1 and 2; the second retains position 3. Hence $\bar K=(1.5,3)^\top$ and $\bar V=(15,30)^\top$ .

Full attention gives

y=\frac{10e^1+20e^2+30e^3}{e^1+e^2+e^3} \approx25.7521.

Compressed attention gives

\bar y=\frac{15e^{1.5}+30e^3}{e^{1.5}+e^3} \approx27.2636.

The compressed output is about 1.5115 higher. Averaging two keys before exponentiation does not preserve the original softmax normalisation, and averaging their values does not preserve every query's preferred mixture. This projection was not learned, and this example supplies no approximation-error bound for Linformer.

Three scalar key-value positions with keys 1, 2, 3 and values 10, 20, 30 are compressed to keys 1.5, 3 and values 15, 30. For query 1, full attention weights are 0.0900, 0.2447, 0.6652 and compressed weights are 0.1824, 0.8176. The outputs are 25.7521 and 27.2636.

Illustrative fixed sequence projection, not a trained Linformer. The output bars share a linear scale in arbitrary value units.

Which matrix has low rank?

For the unmasked compressed head above,

\bar Y=(BF)V

. The effective position-mixing matrix

BF

has rank at most

r

. It need not be a probability matrix: general learned

F

can introduce negative coefficients or alter row sums.

A different claim—“the logits have low rank, so the softmax matrix does too”—is false. Consider

L=\begin{pmatrix}0&0\\0&1\end{pmatrix}.

It has rank one. Row-wise softmax gives a first row $(1/2,1/2)$ and a second row $(1/(1+e),e/(1+e))$ . Its determinant is $(e-1)/(2(1+e))>0$ , so the softmax matrix has rank two. Low rank before a nonlinear transformation does not establish low rank afterward.

Linformer's learned projection is also different from taking the best truncated SVD of a fixed attention matrix. The SelfMadeQuant lesson on the Eckart–Young theorem develops that fixed-matrix optimality result; the full lesson requires membership.

Two conditions behind the saving

Hold the compressed dimension fixed. With feature widths fixed, projection, score formation and value aggregation cost

O(nr)

, with

O(nr)

attention coefficients rather than

O(n^2)

. This is linear in

n

only if

r

stays fixed. Increasing

r

with sequence length changes the scaling. A smaller coefficient array alone does not establish an end-to-end speedup.

Preserve the information available at each query time. A compressed slot can mix past and future observations. In the example, at time 1 even retaining only slot A exposes

(v_1+v_2)/2

, which contains future

v_2

. Masking other compressed slots cannot undo that mixing. Forecasting needs prefix-safe compression or another explicitly causal construction, as well as checks for look-ahead bias in the input pipeline. Bidirectional encoding of a transcript already available at the decision time has a different information boundary.

Sources

Wang et al. (2020), Linformer: Self-Attention with Linear Complexity, §§2.1, 3–4: ordinary attention, low-rank approximation motivation and sequence projection.
Facebook Research, fairseq Linformer example and MultiheadLinearAttention: compression parameters, feature projection and forward computation. Archived implementation, checked 1 November 2026.

Compress positions, not features

A compression you can calculate

Which matrix has low rank?

Two conditions behind the saving

Sources

Related terms

Look-ahead bias

Matrix rank

Softmax and log-sum-exp