What neural-network universality guarantees: continuous targets on compact domains, arbitrary width and uniform error, with a three-ReLU construction and practical limits.
Universal approximation theorem
The universal approximation theorem says that, under specified activation and architecture conditions, sufficiently wide feedforward neural networks can approximate every continuous function on a compact input domain as closely as desired. For a quant building a surrogate for a continuous pricing or risk calculation over a bounded set of states, it establishes expressive capacity. It does not establish that training will find the approximation or that a market forecast will work.
Read the quantifiers
One useful version fixes a continuous activation σ:R→R that is not a polynomial on the whole real line. Let K⊂Rd be compact and let f:K→R be continuous. For every ε>0, there are a finite width m, input weights wj, biases bj and output weights aj such that
g(x)=j=1∑majσ(wj⊤x+bj),x∈Ksup∣f(x)−g(x)∣<ε.
This is a single hidden layer with a linear output. The parameters are freely chosen, and width may grow with the target and requested accuracy. The supremum is the uniform error: the bound holds everywhere in the selected domain, rather than only at observed inputs. Biases are part of the architecture; dropping them changes the function family. This continuous-activation version follows from Leshno, Lin, Pinkus and Schocken's result.
ReLU, σ(z)=max(0,z), is continuous and globally nonpolynomial, even though each of its two pieces is linear. It meets these assumptions. Cybenko's earlier theorem uses a continuous sigmoidal activation on the unit cube; ReLU is not sigmoidal, so its inclusion needs the broader result.
Three ramps, one tent
A concrete construction makes the role of width and signed output weights visible. Write R(z)=max(0,z) and take the dimensionless synthetic response
f(x)=R(x)−2R(x−1)+R(x−2),x∈[−1,3].
There are three hidden units in parallel, each with input weight 1. Their biases are 0,−1,−2, and the output weights are 1,−2,1. At each successive kink, the slope changes: first from zero to one, then to minus one, then back to zero. Thus
f(x)=⎩⎨⎧0,x,2−x,0,x≤0,0<x≤1,1<x<2,x≥2.
Input x
−1
0
0.5
1
1.5
2
3
Output f(x)
0
0
0.5
1
0.5
0
0
Three hidden ReLU ramps with kinks at 0, 1 and 2 are combined with output weights 1, minus 2 and 1. Their dimensionless tent output is zero at minus 1, 0, 2 and 3, one half at 0.5 and 1.5, and one at 1.
Original synthetic construction with fixed parameters. The hidden ramps grow individually; their signed sum rises to one at x = 1 and returns to zero at x = 2. No parameters were learned.
Here the representation is exact because the target was chosen to be a simple piecewise linear function. This is an illustration of how units combine, not a proof that arbitrary continuous functions in any dimension can be approximated.
What “universal” leaves unresolved
The order is target and tolerance first, suitable network second. It does not promise that one fixed small network achieves every accuracy. Nor does the existence statement provide a useful width bound, sample requirement or training algorithm. A multilayer perceptron may have adequate capacity while an optimiser fails to find useful parameters; a low training loss may still reflect overfitting.
The domain and error criterion matter in finance. Continuity of the target must be justified: a digital payoff has a jump, so this continuous-target uniform theorem does not cover it across that jump. Approximation in an average-error norm is a different claim. A value-error bound also gives no derivative-error bound, so an accurate pricing surrogate need not produce accurate Greeks. Finally, control on K supplies no guarantee outside K, including market states absent from the chosen approximation domain.
For the network architecture behind the statement, Neural Networks: Architecture and Non-Linear Models develops hidden layers, affine collapse and activation functions. The preview is public; the full lesson requires membership.
