Skip to content

Production trading system

How a production trading system separates order state, fills, ledger reconciliation and risk controls, including duplicates, uncertain acknowledgements and recovery.

A production trading system is the software and operating process that turns trading decisions into controlled orders, records what actually traded, and keeps positions, cash and risk consistent while markets and components keep changing. For a quant, producing a plausible signal is only part of the job: the system must also explain its exposure after a partial fill, a lost connection or a restart.
An order management system tracks order lifecycles; an execution management system helps carry out execution decisions. A production system connects these responsibilities to market data, strategy logic, risk checks, ledgers and operators. Its boundary can span several services and vendors. “Production” describes how it is operated with real consequences; it does not imply a particular programming language, architecture or speed. A low-latency trading system adds demanding timing requirements to this wider correctness problem.

Keep four different truths separate

ResponsibilityQuestion it answersA dangerous shortcut
Order stateIs the order pending, working, partly filled, filled, rejected or canceled?Treating “sent” as “accepted”
Execution recordsWhich individual fills occurred, for how much, and at what price?Booking the same execution twice
Cash and position reconciliationDoes the internal ledger agree with independent broker or clearing records for the same scope and cutoff?Trusting a green order-status label as proof of balances
Risk controlsWhat further activity is permitted given positions, working orders and uncertainty?Checking only the last known filled position
Status names and transition rules depend on the API and venue. FIX distinguishes the event reported by ExecType from the current order state in OrdStatus; pending cancellation does not mean canceled. See the FIX order-state specification, especially its introduction and cancellation scenarios. Interactive Brokers likewise documents duplicate status callbacks and fills during pending cancellation.

One order, three messages, two fills

Take a fictional buy order for 5 shares, starting with USD 1,000 cash and no shares. F1 reports a purchase of 3 shares at USD 100 per share. An identical F1 arrives again. F2 then reports 2 shares at USD 101 per share. There are no fees, financing costs, taxes, other trades or cash flows. “Cash” below is a simplified trade-date ledger balance, not a claim about settled cash or broker buying power.
A fictional five-share buy order: start with cash USD 1000 and position zero; F1 buys three at USD 100, leaving cash 700 and two shares unfilled; duplicate F1 changes nothing; F2 buys two at USD 101, leaving cash 498 and position five. Unique fill cost is USD 502; wealth marked at USD 101 is USD 1003.
A fictional five-share buy order: start with cash USD 1000 and position zero; F1 buys three at USD 100, leaving cash 700 and two shares unfilled; duplicate F1 changes nothing; F2 buys two at USD 101, leaving cash 498 and position five. Unique fill cost is USD 502; wealth marked at USD 101 is USD 1003.
Three received messages represent two unique fills. The duplicate changes neither the ledger nor the simplified order state. Figures are fictional and exclude fees.
The purchase cost is 3 × 100 + 2 × 101 = USD 502. Ending cash is 1,000 − 502 = USD 498 and the position is 5 shares. At a USD 101 mark, wealth is 498 + 5 × 101 = USD 1,003. The USD 3 increase is an unrealised mark-to-market gain before costs, not realised trading profit. A matching final share count alone would not establish that cash was booked correctly.

Here is a deliberately small offline Python example, executed with Python 3.14.4. The input contains only the validated fictional buy fills above; whole-dollar integer arithmetic is sufficient for this example.

events = [("F1", 3, 100), ("F1", 3, 100), ("F2", 2, 101)] seen = {} cash, position = 1000, 0 order_quantity = 5 for fill_id, quantity, price in events: details = (quantity, price) if fill_id in seen: if seen[fill_id] != details: raise ValueError("conflicting fill identifier") continue seen[fill_id] = details cash -= quantity * price position += quantity state = "FILLED" if position == order_quantity else "PARTIAL" wealth = cash + position * 101 print(state, position, cash, len(seen)) # FILLED 5 498 2 print(wealth) # 1003
This illustrates idempotency: applying the identical fill again has no additional economic effect. The dictionary also detects inconsistent data under the same identifier. Its identity rule is fictional and limited to this one order. Real execution identifiers need their documented account, session or venue scope, and corrections and trade cancellations need explicit handling; IBKR's execution documentation gives one concrete correction convention.
This is not durable or restart-safe infrastructure. Restarting loses seen, cash and position. The example has no concurrent writers, persistence, order submission, acknowledgement processing or reconciliation. Its two-state expression does not implement a real order state machine.

When the connection fails

A rejected new order with no fills creates no position from that order. Record the reason and decide whether a corrected request is allowed; repeatedly resending an invalid request is not recovery. Rejection of a cancel request has a different meaning: the original order may still be working or may already have filled.

A missing acknowledgement is more difficult. The destination may have accepted the order before the response was lost. Mark its outcome as uncertain, retain its identity, and use the protocol's supported status and execution recovery procedure. Sending a fresh order blindly can double the intended trade. Until uncertainty is resolved, risk checks need a conservative treatment of the potentially live quantity. A successful local send or a restored socket does not settle the question.

Recovery needs durable records of intents, identifiers, accepted events and processing progress. Within the local ledger, recording a fill's identity and applying its cash and position effects should succeed or fail together, with concurrency controls that prevent two workers booking it. Database transactions provide an all-or-nothing local mechanism; they do not make a broker order submission and a database update one atomic action. Restart procedures must recover the relevant order and execution history, respect the provider's replay limits, and reconcile before resuming under declared rules. Corrections, fees and settlement differences require their own ledger treatment.

Controls must lead to actions

A risk limit can block a proposed order before routing, using exposure that includes working orders and unresolved submissions. Other checks can reject stale market data, invalid sizes or prices, and excessive submission rates. These controls constrain activity; they do not guarantee a profitable strategy or eliminate execution risk.

Observability means retaining enough evidence to understand operation: correlated order and execution identifiers, event and receipt times, data age, rejects, unreconciled balances and the reason a control fired. Alerts need an owner, an escalation rule and a defined response.

Operators need tested controls to pause new orders, request cancellation, restrict a strategy and authorise resumption. Pausing new orders does not cancel existing ones; cancellation requests need confirmation, and neither action reverses completed fills. A kill control's scope must be explicit, including what happens when connectivity is unavailable. Recovery exercises and incident records are part of operating the system.

For a focused continuation, Monitoring and incident handling develops monitoring contracts, named owners and freeze actions in a paper-trial setting. The link is a public preview; the full lesson requires membership. It teaches operational decisions rather than a broker connectivity implementation.

Sources

Last updated
  • trading systems
  • order lifecycle
  • reconciliation
  • risk controls
  • software engineering
← Back to the glossaryBack to top ↑