Trust-region policy optimisation: Definition & Examples

Trust-region policy optimisation (TRPO) is a reinforcement-learning method that improves a stochastic policy while constraining how much its action distribution changes. For a quant training an execution policy in a simulator, this addresses a specific failure: an apparently helpful update can change trading behaviour enough to make the data used to judge that update misleading.

It builds on policy gradients, but measures the proposed step in probability distributions rather than just parameter coordinates.

The update has two tests

An advantage measures how much better an action is than the old policy's usual outcome at that state. TRPO estimates a surrogate gain using old-policy data:

S(\theta)=\mathbb{E}_{s\sim d_{\mathrm{old}},\,a\sim\pi_{\mathrm{old}}} \left[\frac{\pi_\theta(a\mid s)}{\pi_{\mathrm{old}}(a\mid s)}\widehat A_{\mathrm{old}}(s,a)\right].

Here $d_{\mathrm{old}}$ is the normalised discounted state-visitation distribution; hats indicate estimated advantages. The probability ratio reweights actions sampled under the old policy. It requires old-policy support wherever the candidate puts mass.

Using the original paper's old-to-new KL convention, the practical constraint is

\mathbb{E}_{s\sim d_{\mathrm{old}}} \left[D_{\mathrm{KL}}\!\left(\pi_{\mathrm{old}}(\cdot\mid s)\,\|\,\pi_\theta(\cdot\mid s)\right)\right]\leq\delta.

The surrogate holds state weights fixed, although a new policy generally visits different states. That is why improving it alone is insufficient. The paper's theoretical improvement bound controls the maximum divergence across states; practical TRPO replaces that with an average estimated from sampled states. The original guarantee therefore does not automatically carry over to every practical update. See Schulman et al., Sections 2–5.

A tempting change that is too large

Consider a fictional single-state, two-action calculation, with exact stipulated old-policy advantages. Action 1 has advantage $+1$ reward unit and action 0 has advantage $-1$ . The old policy chooses each with probability $0.5$ , so its expected advantage is zero. A candidate changes the probability of action 1 to $0.6$ :

Action	Old probability	Candidate probability	Probability ratio	Old advantage
1	0.50	0.60	1.20	+1
0	0.50	0.40	0.80	−1

Its surrogate gain is

0.5(1.2)(1)+0.5(0.8)(-1)=0.20.

But, using natural logarithms,

D_{\mathrm{KL}}(\mathrm{old}\|\mathrm{new}) =0.5\log\frac{0.5}{0.6}+0.5\log\frac{0.5}{0.4} \approx 0.020411\ \text{nats}.

For an illustrative limit of $\delta=0.01$ , the candidate fails despite a probability change of only ten percentage points. A smaller candidate, probability $0.55$ , gives surrogate gain $0.10$ and KL $0.005025$ nats, satisfying that limit. The threshold is chosen to teach the constraint, not recommended as a trading or training setting.

Fictional two-action policy with old action-1 probability 0.50. The KL curve in nats is below the illustrative 0.01 limit at new probability 0.55, where KL is 0.00503 and surrogate gain is 0.10. At probability 0.60, KL is 0.02041 and surrogate gain is 0.20, so that candidate fails the limit.

Original exact Bernoulli calculation. Shading marks probabilities satisfying the illustrative KL limit; a larger surrogate gain does not make an infeasible candidate acceptable.

This is an exact surrogate comparison, not a run of the full TRPO algorithm. It contains no changing state distribution, estimated critic or transaction costs. The gain is in stipulated reward units, not portfolio return. KL is asymmetric: reversing its arguments changes the calculation.

How a large policy is updated

The implementation uses a linear surrogate and a quadratic local KL approximation. The latter gives the Fisher information matrix. A conjugate-gradient solve finds an approximate direction without forming its inverse. Backtracking then shrinks a trial step until the sampled nonlinear surrogate improves and the sampled KL constraint passes; if no acceptable step is found, the policy can stay unchanged. This extra check matters because local approximations can misjudge a finite step. Spinning Up's derivation and pseudocode explain that workflow.

Passing these checks constrains an update within the sampled training problem. It cannot establish that an execution simulator has realistic fills, that its reward captures inventory risk, or that tomorrow's market follows its dynamics. TRPO offers no guarantee of trading profit, and sampled practical updates need not improve true expected reward monotonically.

For the gradient underneath this method, read the policy-gradient section of the SelfMadeQuant lesson Deep RL and Policy Gradients at the Survey Level (public preview; full lesson requires membership).

Sources

Schulman et al., Trust Region Policy Optimization, Sections 2–6 and Appendix C: surrogate, maximum-versus-average KL and constrained optimisation.
OpenAI Spinning Up, Trust Region Policy Optimization, Key Equations and Pseudocode: local approximation, conjugate gradients and backtracking. Its displayed KL direction differs from the convention used in the worked example above.