Skip to content

Hierarchical reinforcement learning

Reinforcement learning with policies that choose extended skills, illustrated by a three-step option backup and the difference between option and episode termination.

Hierarchical reinforcement learning (HRL) organises reinforcement learning into levels of decisions: a higher-level policy chooses a skill or subtask, and a lower-level policy selects the actions that carry it out. For a quant execution system, this could separate choosing an inventory-reduction objective from deciding which orders to submit as market conditions change.

The useful abstraction is a course of behaviour with duration, rather than another layer in a neural network. HRL can be implemented with or without deep networks.

A skill has a starting rule and an ending rule

The options framework makes this precise. An RL option is not a financial derivative. A Markov option has three components:
ComponentMeaningIllustrative execution design
Initiation set IoI_oStates where the option may startAn outstanding sell quantity exists and execution is permitted
Internal policy πo(a∣s)\pi_o(a\mid s)Distribution over primitive actions while it runsChoose an admissible order size and type using current inventory and market observations
Termination rule βo(s)\beta_o(s)Probability of ending the option upon reaching a stateEnd when the subtarget is reached or the execution mode is no longer suitable
In call-and-return execution, the higher-level policy selects an available option and follows its internal policy until it terminates; then it chooses again. The internal policy can react to each new state: the skill need not be a fixed sequence of orders. See the Option-Critic paper's preliminaries and execution model.
A time-based stopping rule requires an appropriate clock in the state, or an option that can use its history since initiation. A state description does not acquire the Markov property merely because a hierarchy uses it.

Three steps are not one discount period

At option boundaries, the process is semi-Markov: elapsed duration is part of the transition. For a completed option lasting τ\tau primitive steps, a sample backup target is
y=∑j=0τ−1γjrt+j+1+γτV(st+τ).y=\sum_{j=0}^{\tau-1}\gamma^j r_{t+j+1} +\gamma^\tau V(s_{t+\tau}).
Here γ\gamma is a discount per primitive step and VV is the estimated continuation value under the chosen higher-level continuation policy. For Q-learning control, that continuation is the largest estimated value among available next options. This duration-sensitive backup follows Sutton, Precup and Singh, §3.2.
Take an original synthetic trajectory in dimensionless reward units. An option lasts three steps, pays successive rewards 1, 2, 3, and reaches a nonterminal state whose estimated continuation value is 10. Use γ=0.9\gamma=0.9:
Within-option reward=1+0.9×2+0.92×3=5.23,Continuation contribution=0.93×10=7.29,y=5.23+7.29=12.52.\begin{aligned} \text{Within-option reward}&=1+0.9\times2+0.9^2\times3=5.23,\\ \text{Continuation contribution}&=0.9^3\times10=7.29,\\ y&=5.23+7.29=12.52. \end{aligned}
A synthetic option spans three primitive steps with rewards 1, 2 and 3. At a 0.9 per-step discount, the within-option reward is 5.23. A nonterminal continuation value of 10 contributes 0.9 cubed times 10, or 7.29, giving a backup target of 12.52. At a true episode terminal state, the target is 5.23.
A synthetic option spans three primitive steps with rewards 1, 2 and 3. At a 0.9 per-step discount, the within-option reward is 5.23. A nonterminal continuation value of 10 contributes 0.9 cubed times 10, or 7.29, giving a backup target of 12.52. At a true episode terminal state, the target is 5.23.
Original synthetic timeline in dimensionless reward units. The option ends after three steps; the task continues unless the reached state is genuinely terminal.
Discounting the continuation only once would produce 14.23, overstating this target. The clock advances for every primitive step, even while the higher-level policy is not choosing.
If the third transition genuinely ends the episode, continuation is zero and the target is 5.23. Option termination alone leaves the next decision's value in place. These are estimated reward targets, not realised trading returns; the learning discount is not automatically an interest rate.
The lesson Markov Decision Processes and Value/Policy Iteration supplies the finite-state Bellman foundation. Its preview is public; full teaching requires membership.

The hierarchy still has to earn its complexity

Options may be specified by a designer or learned. Option-Critic learns internal policies and termination functions alongside a policy over options, without requiring extra subgoal rewards. This is one architecture, not the definition of HRL. Its original experiments do not establish a financial trading advantage. Bacon, Harb and Precup (2017)
In the illustrative execution design, both levels must agree on the objective. A lower level rewarded only for fast completion could consume more spread than a higher level intended. Count transaction costs consistently across levels, retain hard inventory and participation limits, and state who can interrupt a skill when those limits bind. A hierarchy is no substitute for those controls.

Nor does a reusable skill guarantee useful exploration or faster learning. Poor options can restrict the available behaviours; training their internal policies also consumes data. In finance, evaluation must additionally test whether the lower-level fills and market impact assumptions remain credible when the higher level changes the order flow it generates. A convincing hierarchy needs comparison with a simpler policy under the same information, costs and constraints.

Sources

  • Sutton, R. S., Precup, D. and Singh, S. (1999), Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning, Artificial Intelligence 112, pp. 181–211. §2 defines options and distinguishes Markov from history-dependent options; §3 gives the SMDP connection; §3.2 gives the duration-sensitive learning backup.
  • Bacon, P.-L., Harb, J. and Precup, D. (2017), The Option-Critic Architecture, AAAI 31, pp. 1726–1734; linked author preprint v2, 3 December 2016. Introduction, Preliminaries and Notation, and Learning Options read for call-and-return execution and learning the internal policies and termination functions.
Last updated
  • reinforcement learning
  • temporal abstraction
  • options framework
  • semi markov decision process
  • execution
Keep connecting the ideas
← Back to the glossaryBack to top ↑