Learning a decision policy by predicting demonstrator actions, with a categorical likelihood example and the state-distribution mismatch that can derail trading imitation.
Behaviour cloning
Behaviour cloning learns a decision policy by treating a demonstrator's actions as supervised-learning labels. For a quant, this could mean predicting an execution trader's next order from the information available when that order was chosen. The immediate training target is agreement with the demonstrator; profitable or optimal trading requires separate evidence.
The input might contain inventory, spread, recent order flow and time remaining. The label might be an order type, a signed quantity or a participation rate. A policy maps that input to an action or an action distribution. It need not use a neural network: classification and regression models can both perform behaviour cloning. Levine's Berkeley lecture develops this supervised view of imitation learning.
Fitting the action, not the reward
For discrete actions, a common objective is maximum likelihood estimation. Given N demonstration pairs (xi,ai), fit policy parameters θ by minimising
LBC(θ)=−N1i=1∑Nlogπθ(ai∣xi).
Here xi is the observation or recorded history available to the learner, ai is the demonstrated action, and πθ(ai∣xi) is the probability assigned to it. This is cross-entropy training for categorical actions. Continuous quantities need an appropriate regression loss or action-density model; an average of two valid but very different orders may be an unsuitable order itself.
Consider four fictional demonstrations with the same recorded input A: buy, buy, sell, buy. Assume only buy and sell are allowed, with one free, unregularised probability p for buying at A. The likelihood is p3(1−p), so its maximum occurs at
p^=43=0.75.
The fitted mean negative log-likelihood, using natural logarithms, is
−43ln(0.75)−41ln(0.25)=0.562335 nats per demonstration.
This is an in-sample fit to four labels, not a population probability established with precision. Different actions at the same recorded input can reflect demonstrator randomisation, omitted information or mixed objectives. They do not establish that buying is the better trade. Sampling from this fitted policy buys with probability 0.75; choosing its most probable action always buys at A. Those deployment rules produce different behaviour.
Fictional demonstrations at observed state A contain three buys and one sell. The categorical maximum-likelihood fit assigns buy probability 0.75 and sell probability 0.25, with mean negative log-likelihood 0.562335 nats. A separate schematic shows how a learner action can lead to state C while the demonstrator path visits state B; state C has no demonstration label in this sketch.
Original fictional action-fit calculation and schematic state-distribution mismatch. The path sketch assigns no transition probabilities or investment returns.
For the likelihood mechanics, Logistic Regression and Maximum Likelihood derives the binary objective and gradient. The preview is public; the full lesson requires membership.
Why a good action predictor can be a poor controller
The demonstrator generated the training states. Once the learner acts, its own decisions help generate subsequent states. A slightly different fill can change inventory, which changes the next appropriate order. The learner may then encounter combinations that the demonstrator rarely visited, make further mistakes there and drift farther from familiar states. This feedback creates compounding errors, even if held-out action prediction looks good on demonstration trajectories.
Ross, Gordon and Bagnell formalise this mismatch between demonstrator-state and learner-state distributions. Their DAgger procedure runs the learner, obtains the expert's action labels for the states visited, aggregates those labelled examples and retrains. It requires access to an expert who can label those new states; a fixed historical trading log cannot supply the missing counterfactual decisions. Its theoretical guarantees also depend on the paper's assumptions, rather than establishing trading profitability.
What a trader log must preserve
An imitation dataset needs more than prices and executed trades. Record the decision timestamp and the inputs actually available then. A submitted order, its later fill and its cancellation are different events: using a realised fill as though it were the earlier decision can mislabel the action. Including subsequent market data introduces look-ahead bias.
The learner also needs the information that made the demonstrator's choice intelligible. If a trader saw an unrecorded client instruction or risk limit, the same recorded market features can correspond to different intended actions. Conditioning only on those features estimates the mixture visible in the logs, not the missing decision rule.
Use later held-out trajectories to assess imitation, then assess the resulting sequential policy under explicit fill, cost and inventory assumptions. Agreement metrics answer a different question from net performance. Behaviour cloning can provide a reproducible baseline or an initial policy for reinforcement learning. Offline reinforcement learning also uses logged data, but seeks a reward objective rather than simply reproducing logged actions.
The historical roots are in control, not finance: Pomerleau's ALVINN paper trained a network on simulated road scenes and desired steering outputs. Its specific training setup should not be confused with evidence that trader logs contain enough information to reproduce a trading policy.
