Reinforcement learning (RL) is a learning setup in which an agent learns a policy—a rule for choosing actions—from experience with an environment and its rewards, seeking to maximise expected cumulative reward. In quantitative finance, the decision might be how much of an order to execute now: today's action changes the remaining inventory, the costs paid and the choices available later.
The defining problem is sequential decision-making. A price forecast alone does not specify what to trade, and a profitable simulated policy does not establish profitable live execution.
Specify the decision before choosing the algorithm
An execution task makes the ingredients concrete. These are modelling choices, rather than a universal market specification:
| Ingredient | Possible execution specification |
|---|---|
| State or observation | Remaining quantity, time to the deadline, current spread, visible depth and recent fills, all available at the decision time |
| Action | An allowed order size and type, or waiting, subject to position, participation and venue constraints |
| Environment | The market or simulator that determines fills, prices and the next observation after an action |
| Reward | Negative execution cost relative to a stated benchmark, including fees and a stated treatment of inventory risk or unfinished quantity |
| Policy | A rule mapping the available information to an action or a distribution over actions |
A reward is a numerical objective supplied by the problem designer. It need not be profit. A reward that omits transaction costs can favour excessive trading; one that ignores an execution deadline can reward waiting indefinitely. Hard constraints should restrict admissible actions, rather than relying solely on a penalty to make violations unattractive.
A Markov decision process uses a state with the Markov property: it contains the information needed for the conditional distribution of the next state and reward, given the action. A compact collection of market features is not automatically such a state. Hidden liquidity and other unobserved influences may make an observation only a partial description of the environment.
The owned lesson Markov Decision Processes and Value/Policy Iteration develops finite-state Bellman recursions and planning with transition models. The full lesson requires membership; its public preview gives the purpose and outline.
Learning about later consequences
In supervised learning, a training example supplies an observed target, such as a later return. RL instead evaluates actions through rewards and their later consequences. Rewards do not generally identify the best action for each state: an apparently costly action now may improve what is achievable later.
Q-learning is one value-based RL algorithm. In its tabular form, it maintains an estimate for each state–action pair, aiming to learn the expected discounted cumulative reward from taking that action and acting optimally afterwards. After a transition from to a nonterminal state , with observed reward , its one-step update is
Here contains the admissible next actions, is the learning step size and weights later rewards. This learning discount is a modelling choice; it is not automatically a market interest rate. The target combines the observed immediate reward with a current estimate of continuation value. Using an estimate to update another estimate is called bootstrapping.
Take a synthetic transition with dimensionless reward units. Let the old estimate be 1, the observed reward 2, the discount 0.9, the highest admissible next-action estimate 4, and the step size 0.5. Then
Original synthetic one-transition calculation in dimensionless reward units. The update changes an estimate; it is not a realised trading return.
If the next state is genuinely terminal, there are no subsequent rewards in that episode: use a zero continuation contribution. With the same old estimate, reward and step size, the target is 2 and the updated estimate is 1.5. Merely stopping a data sample or imposing an artificial training cutoff does not by itself make the economic task terminal.
Neither calculation proves that the learned values are accurate or that the chosen action is best. Tabular convergence results require assumptions about the environment, repeated state–action coverage and the step-size sequence. A single update—and a fixed step size in this illustration—does not establish those conditions. See the original Watkins–Dayan Q-learning paper, especially §§2–3.
RL also includes policy-gradient methods that optimise a parameterised policy directly, and actor–critic methods that combine policy and value learning. Q-learning is an example, not a definition of the whole field. The Stanford CS234 materials cover Q-learning, policy gradients and offline RL as distinct topics.
The environment is part of the result
Learning poses an exploration problem: gathering information may require actions that currently look worse. Live financial exploration consumes capital and can incur losses or breach constraints. Training in a simulator avoids some direct market costs, but the agent can learn to exploit unrealistic fills, absent market impact or a weak penalty for uncompleted orders.
Historical observations also do not reveal every counterfactual. A replay showing what followed a small historical order does not establish what would have happened after a much larger order. Offline RL learns from previously collected experience; its reliability depends on which state–action situations that experience covers and the assumptions used to evaluate alternatives. RL does not require unrestricted trial and error in a live account.
Evaluate a frozen training and tuning procedure on separate later periods, with decision-time information, realistic costs and explicit risk limits. Compare it with credible simpler execution rules. Reusing the evaluation period to redesign the reward, simulator or policy introduces overfitting, just as repeated strategy selection does in backtesting. Continued learning can change a policy; it provides no guarantee of successful adaptation, robustness to market changes or future profit.
Sources
- Sutton, R. S. and Barto, A. G., Reinforcement Learning: An Introduction, local “Second edition, in progress” text, ©2014, 2015; §§1.1 and 1.3, pp. 2–3 and 7–8, and §6.5, pp. 157–158. Read for the learning setup, policy/reward/value distinction, one-step Q-learning and terminal-state convention. This local text is a prepublication version.
- Watkins, C. J. C. H. and Dayan, P. (1992), Q-learning, Machine Learning 8, pp. 279–292, §§2–3. Original algorithm and finite tabular convergence assumptions.
- Stanford University, CS234: Reinforcement Learning and its lecture materials, Winter 2026. Read for sequential decision formulation, exploration and the distinction between learning approaches. Checked 27 October 2026.
- López de Prado, M. (2018), Advances in Financial Machine Learning, §§11.2–11.4. Local text read for historical simulation limits, information timing, transaction costs and selection risk. The execution specification and numerical illustration here are original.