HiPPO and Legendre memory units: Definition & Examples

HiPPO (high-order polynomial projection operators) is a framework for compressing an evolving signal's past into polynomial coefficients. A Legendre Memory Unit (LMU) is a recurrent neural-network architecture with a Legendre-based memory cell. For a quant processing a long stream of observations, the attraction is a fixed-size history representation. That alone says nothing about forecast accuracy. Gu et al. (2020) formulate the projection framework; Voelker, Kajić and Eliasmith (2019) introduce the LMU.

What the coefficients remember

Choose a weighting measure over the past and retain

N

polynomial basis functions. The target is the orthogonal projection that minimises weighted squared reconstruction error. For an orthonormal basis

\phi_n^{(t)}

and measure

\mu^{(t)}

c_n(t)=\int f(s)\phi_n^{(t)}(s)\,d\mu^{(t)}(s), \qquad \hat f_t(s)=\sum_{n=0}^{N-1}c_n(t)\phi_n^{(t)}(s).

The choice of measure decides which parts of history matter. HiPPO derives coefficient dynamics that can be discretised into online recurrences. HiPPO, §§2.1–2.4.

A two-number memory you can inspect

Freeze time and use a synthetic, dimensionless history $f(u)=u$ on $[0,1]$ , weighted uniformly. Here $u$ runs from the beginning to the end of this fixed interval. Take

\phi_0(u)=1, \qquad \phi_1(u)=\sqrt{3}(2u-1).

These functions have unit squared integral and zero cross-inner-product. Their coefficients are

\begin{aligned} c_0&=\int_0^1u\,du=\frac12,\\ c_1&=\int_0^1u\sqrt{3}(2u-1)\,du =\frac{\sqrt{3}}{6}\approx0.288675. \end{aligned}

The first coefficient carries the mean level. The second carries the linear variation around it:

\hat f(u)=\frac12+\frac{\sqrt{3}}6\sqrt{3}(2u-1)=u.

At $u=0.25$ the contributions are $0.5-0.25=0.25$ ; at $u=0.75$ they are $0.5+0.25=0.75$ . Keeping only the mean instead gives integrated squared error $\int_0^1(u-0.5)^2du=1/12$ . Keeping both gives zero error.

Static uniform-weight projection of f(u)=u on [0,1]: mean coefficient c0=0.5, linear coefficient c1 approximately 0.288675, mean contribution 0.5 and linear contribution u minus 0.5. Their sum reconstructs u exactly, including 0.25 and 0.75; no dynamic HiPPO update or LMU training is simulated.

Original dimensionless example. The two component curves sum to the input; the exact fit is possible because this input is already a degree-one polynomial.

This exact reconstruction works because the input already lies in the retained polynomial space. Two coefficients cannot preserve an arbitrary history. This is a static projection illustration, not a simulation of a HiPPO transition or trained LMU.

The interval changes as observations arrive

Translated Legendre (LegT) weights a fixed recent window

[t-\theta,t]

. Its update approximates the unavailable departing observation using the compressed history. Scaled Legendre (LegS) weights all of

[0,t]

uniformly and rescales the basis as that interval grows; its continuous dynamics contain a

1/t

factor. Thus “Legendre memory” does not specify one universal lookback or transition matrix. HiPPO, §2.3, §3 and Appendix D.1.

Where the LMU fits

The original LMU predates HiPPO. Its linear memory obeys

\theta\dot{\mathbf m}(t)=A\mathbf m(t)+Bv(t),

where

\theta

sets a sliding-window timescale. Discretisation gives a recurrent update. The architecture couples this memory to a learned nonlinear hidden state and learned encoders; the signal

v(t)

written to memory can itself be a learned combination of inputs and prior states. The paper allows training memory parameters, but does not require it in its experiments. A fixed polynomial projection is therefore only part of the architecture. LMU, §2, equations (1)–(7), and §3 initialisation.

HiPPO recovers the LMU's core update through its LegT construction, with the appropriate basis scaling. LegS is a different history-weighting choice. HiPPO, §2.3 and Appendix D.1.

For financial sequence work, separate reconstruction quality from predictive relevance: retaining a past path accurately does not establish that it contains a tradable signal. Define the time coordinate explicitly—event number and elapsed clock time weight market history differently. At a forecast origin, update memory only with information already available then; fitting encoders and choosing memory size also belong inside the training period.

For the mathematical foundation, the SelfMadeQuant lesson Orthogonal Projections and Minimization Problems develops closest-point geometry. Recurrent Networks for Structured Data explains the state and gradient path of a recurrent neural network. Both offer public previews; full lessons require membership.

Sources

Gu, Dao, Ermon, Rudra and Ré (2020), HiPPO: Recurrent Memory with Optimal Polynomial Projections, §§2–3 and Appendix D.1.
Voelker, Kajić and Eliasmith (2019), Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks, §2 and §3 initialisation.

What the coefficients remember

A two-number memory you can inspect

The interval changes as observations arrive

Where the LMU fits

Sources

Related terms

Orthogonal projection