Deep kernel learning learns a neural-network transformation of the inputs together with a kernel applied to that transformation. In its standard Gaussian-process form, the network weights and kernel parameters are trained through the GP marginal likelihood. For a quant building a nonlinear regression model, this means learning which market observations should be treated as similar, rather than fixing similarity in the original feature coordinates. Wilson et al. (2016), §4 gives the defining construction.
The network changes the kernel's geometry
Write the neural feature map as and the base kernel as . The effective kernel is
The network can bring inputs together or push them apart in feature space. A stationary base kernel can therefore produce nonstationary similarity in the original input space: equal raw displacements need not correspond to equal feature displacements. Input warping as a covariance construction is discussed in GPML, §4.2.
A minimal calculation isolates that mechanism. Take the RBF kernel with unit variance and unit length scale,
and the deterministic map . For the unitless inputs and , the raw distance is one and the feature distance is two:
| Representation supplied to the base kernel | Similarity |
|---|---|
| Raw inputs | |
| Mapped inputs |
Original deterministic illustration, not a fitted deep network. Scaling the feature distance changes similarity while leaving the base RBF formula unchanged.
This linear map is only an arithmetic illustration; no network has been trained. Here it is equivalent to halving the RBF length scale in raw coordinates. A nonlinear learned map can make more elaborate, location-dependent changes.
Kernel validity survives the transformation. If the base kernel is valid, its Gram matrix evaluated at any transformed inputs is positive semidefinite. That is exactly the matrix produced by the composed kernel on the original inputs. For these two points the diagonal entries are one, the off-diagonal entries are , and the eigenvalues are and . Both are positive. The finite example illustrates the general Gram-matrix argument; it does not prove validity on its own. See GPML, §4.1.
What is actually learned
For zero-mean GP regression with independent Gaussian observation noise of variance , let . Training maximizes
This integrates out the latent GP function values. In the usual optimization treatment, it does not integrate out the neural weights. The data-fit and log-determinant terms are coupled through the same covariance matrix; changing the representation changes both. GPML, §5.4 explains this objective.
A frozen embedding followed by a separately fitted GP also defines a composed kernel, but it lacks this joint training step. Deep kernel learning also differs from a deep GP, whose intermediate mappings are themselves stochastic GP layers. The original scalable DKL paper used structured kernel interpolation and inducing points; composing a kernel with a network alone does not make ordinary dense GP computations cheap. Wilson et al., §§2–4.
A GP output is not a calibration certificate
The predictive distribution is conditional on the fitted representation, kernel and likelihood. Uncertainty about optimized network weights is absent from that conditional distribution. Valid covariance matrices ensure a coherent model, not that its intervals achieve their nominal coverage on future observations.
Ober, Rasmussen and van der Wilk (2021) demonstrate that marginal-likelihood-trained DKL can overfit in some settings. For a financial forecasting application, assess predictive coverage on chronologically held-out data and keep feature fitting within each training window. A learned representation can map an unfamiliar market state close to training observations; raw distance alone does not ensure high predictive uncertainty.
For the algebra behind the construction, see The Kernel Trick: Feature Maps and Properties of Kernels. For the conditional prediction formulas, see Gaussian Processes for Regression and Classification. Both offer public previews; the full lessons require membership.
Sources
- Wilson, Hu, Salakhutdinov and Xing (2016), Deep Kernel Learning, §§2–4: composed kernels, joint GP training and the structured approximation used for scalability.
- Rasmussen and Williams (2006), Gaussian Processes for Machine Learning, Chapter 4: covariance validity, Gram matrices and input warping.
- Rasmussen and Williams (2006), Gaussian Processes for Machine Learning, §5.4: regression marginal likelihood and hyperparameter optimization.
- Ober, Rasmussen and van der Wilk (2021), The Promises and Pitfalls of Deep Kernel Learning, §§1–3: deterministic feature extractors, empirical Bayes and observed overfitting.