Skip to content

Kernel principal component analysis

Perform PCA in a nonlinear feature space. An exact quadratic example explains kernel centering, score normalization, covariance eigenvalues and out-of-sample projection.

Kernel principal component analysis (kernel PCA) finds principal components after a nonlinear feature transformation, using kernel evaluations instead of explicitly constructing the transformed features. For a quant compressing market indicators, it can capture variation in nonlinear combinations of those indicators. The resulting components maximise variance in the chosen feature space; they need not be tradable portfolios or the directions most useful for predicting returns.
The kernel trick supplies an inner product k(x,z)=⟨ϕ(x),ϕ(z)⟩k(x,z)=\langle\phi(x),\phi(z)\rangle. Kernel PCA applies ordinary PCA to the centered versions of ϕ(xi)\phi(x_i). A linear kernel recovers ordinary centered PCA. The nonlinear feature-space interpretation is the defining construction in Schölkopf, Smola and Müller's original paper.

Center the features, then find their directions

For nn training observations, form the Gram matrix Kij=k(xi,xj)K_{ij}=k(x_i,x_j). Use a real symmetric positive semidefinite kernel. Let 1\mathbf{1} be the length-nn vector of ones and set
H=I−1n11⊤,Kc=HKH.H=I-\frac{1}{n}\mathbf{1}\mathbf{1}^{\top}, \qquad K_c=HKH.
This subtracts the training mean in feature space, ϕˉ=n−1∑iϕ(xi)\bar\phi=n^{-1}\sum_i\phi(x_i). Centering the input coordinates before applying a nonlinear map generally does not do that job. Appendix B of the original paper derives both training and test centering.
For a positive eigenvalue λj\lambda_j of KcK_c, choose a Euclidean unit eigenvector uju_j. The corresponding unit direction in feature space is
vj=1λj∑i=1nuij(ϕ(xi)−ϕˉ).v_j=\frac{1}{\sqrt{\lambda_j}}\sum_{i=1}^{n}u_{ij}\bigl(\phi(x_i)-\bar\phi\bigr).

The training component scores are therefore

tj=Kcujλj=λj uj.t_j=\frac{K_cu_j}{\sqrt{\lambda_j}} =\sqrt{\lambda_j}\,u_j.

An eigenvector alone is not the score vector: its scale must be restored. Zero eigenvalues supply no nonzero feature direction and cannot be used in this division. Reversing an eigenvector's sign reverses its scores without changing the component subspace.

Three inputs make the normalization visible

Take fictional unitless inputs x=(−1,0,1)x=(-1,0,1) and the homogeneous quadratic kernel

k(x,z)=(xz)2,ϕ(x)=x2.k(x,z)=(xz)^2,\qquad \phi(x)=x^2.

The input mean is already zero, but the feature values are (1,0,1)(1,0,1), with mean 2/32/3. Their centered values form

a=(1/3−2/31/3),Kc=aa⊤=19(1−21−24−21−21).a=\begin{pmatrix}1/3\\-2/3\\1/3\end{pmatrix}, \qquad K_c=aa^{\top} =\frac{1}{9}\begin{pmatrix}1&-2&1\\-2&4&-2\\1&-2&1\end{pmatrix}.

This rank-one matrix has eigenvalues 2/3,0,02/3,0,0. Choose

u=16(1−21).u=\frac{1}{\sqrt{6}}\begin{pmatrix}1\\-2\\1\end{pmatrix}.
Then 2/3 u=a\sqrt{2/3}\,u=a: the scores are (1/3,−2/3,1/3)(1/3,-2/3,1/3), up to a common sign. They are not these values divided by 3\sqrt{3}.
Exact unitless quadratic-kernel example. Inputs minus one, zero and one have scores one third, minus two thirds and one third. The centered Gram matrix has eigenvalues two thirds, zero and zero; the nonzero covariance eigenvalue is two ninths with divisor n or one third with divisor n minus one.
Exact unitless quadratic-kernel example. Inputs minus one, zero and one have scores one third, minus two thirds and one third. The centered Gram matrix has eigenvalues two thirds, zero and zero; the nonzero covariance eigenvalue is two ninths with divisor n or one third with divisor n minus one.
Original deterministic example. The curve shows the fitted score map, not a financial forecast. Equal scores at opposite inputs reveal the sign information discarded by this kernel.
The eigenvalue convention explains the tempting extra factor. Here λ=2/3\lambda=2/3 belongs to the Gram matrix. The feature covariance with divisor nn has nonzero eigenvalue λ/n=2/9\lambda/n=2/9; with divisor n−1n-1, it is λ/(n−1)=1/3\lambda/(n-1)=1/3. The scores stay the same. Software output called “eigenvalues” must be interpreted according to the matrix it diagonalizes; scikit-learn's KernelPCA documentation explicitly identifies the centered kernel matrix.
For a fixed kernel and training set, λj/∑ℓλℓ\lambda_j/\sum_\ell\lambda_\ell measures the fraction of centered feature-space variance retained. In this example, one component retains all such variance, yet −1-1 and +1+1 have become indistinguishable. Retained feature variance does not establish retained information about the original input, economic risk or a prediction target.

A new observation keeps the training origin

For a new input x∗x_*, first evaluate k∗i=k(x∗,xi)k_{*i}=k(x_*,x_i). Center each entry using training quantities:

(k∗c)i=k(x∗,xi)−1n∑r=1nk(x∗,xr)−1n∑r=1nKri+1n2∑r,s=1nKrs.(k_{*c})_i=k(x_*,x_i) -\frac{1}{n}\sum_{r=1}^{n}k(x_*,x_r) -\frac{1}{n}\sum_{r=1}^{n}K_{ri} +\frac{1}{n^2}\sum_{r,s=1}^{n}K_{rs}.

Its score on component jj is

tj(x∗)=k∗c⊤ujλj.t_j(x_*)=\frac{k_{*c}^{\top}u_j}{\sqrt{\lambda_j}}.
For x∗=2x_*=2 in the example, the raw kernel vector is (4,0,4)(4,0,4), the centered vector is (10/9,−20/9,10/9)(10/9,-20/9,10/9), and the score is 10/310/3. Direct feature centering independently gives 22−2/3=10/32^2-2/3=10/3. Re-estimating the origin from the test batch would change the fitted transformation. The official KernelCenterer documentation describes retaining the fitted training-column and overall kernel means.

What the representation does not promise

A kernel PCA score does not automatically have an inverse in input space. Here the score 1/31/3 corresponds to both x=−1x=-1 and x=1x=1. More generally, a projected feature vector may have no exact pre-image at all. An approximate reconstruction requires an additional method; scikit-learn's optional inverse transform learns a pre-image through kernel ridge regression.
The kernel and its input scales also determine which variation looks large. A visually separated embedding is not evidence of future predictive performance. For forecasting, fit scaling, kernel centering and component directions inside each training window; choose kernel settings and component count using a time-respecting validation design, and transform later observations with that fitted mapping. The training-only transformation rule follows the official guidance on preprocessing leakage. Full dense kernel storage also grows as n2n^2, so avoiding explicit features does not eliminate sample-size costs.
To build the linear foundation, the SelfMadeQuant lesson Principal Component Analysis and the SVD View develops the maximum-variance direction. The Kernel Trick develops feature maps and Gram matrices. Both offer public previews; full lessons require membership.

Sources

Last updated
  • kernel methods
  • pca
  • dimensionality reduction
  • feature space centering
Keep connecting the ideas
← Back to the glossaryBack to top ↑