A volatility estimator can be numerically correct and still consume a duplicated close. A risk report can add its inputs correctly and still omit a position whose quantity failed to parse. Data engineering owns the path that prevents those failures from becoming apparently trustworthy results.
Follow a position through the boundaries
The following fictional batch contains signed integer quantities and positive prices in USD per unit. All rows belong to one position snapshot; prices and quantities are already aligned to that snapshot. This is a deliberately small contract, without currency conversion, derivatives valuation or accrued cash flows.
| Raw row ID | Quantity text | Price text, USD per unit | Validation outcome |
|---|---|---|---|
| r1 | 2 | 100 | Accept: position value USD 200 |
| r2 | bad | 50 | Quarantine: quantity is not an integer |
| r3 | 3 | 50 | Accept: position value USD 150 |
These boundaries answer different questions, even when one small program handles several of them:
| Boundary | What it must establish |
|---|---|
| Ingestion | Which source, batch and raw records arrived? Preserve original content and receipt information before changing it. |
| Validation | Do required fields, types, keys and financial rules hold? Send invalid records to quarantine with their original IDs, contents and reasons. |
| Storage | Can the raw snapshot, accepted records and rejects be retrieved under stable identities? Define retention and access controls. |
| Transformation | Which rule turns accepted inputs into outputs? Here it is quantity times price, with units carried into the result. |
| Lineage | Which input snapshot and row IDs, validation rules and transformation version produced this output? |
| Publication | May a consumer use this output, and for which purpose? Attach coverage and quality state to the released result. |
A small validation example
This Python example runs entirely in memory. Its input is the fixed three-row fixture above, with unique IDs and the stated string fields. It illustrates acceptance and quarantine; it does not implement durable storage, source reconciliation or a publication service.
from decimal import Decimal, InvalidOperation
raw = [
{"id": "r1", "qty": "2", "price": "100"},
{"id": "r2", "qty": "bad", "price": "50"},
{"id": "r3", "qty": "3", "price": "50"},
]
accepted, quarantined = [], []
for row in raw:
try:
qty = int(row["qty"])
price = Decimal(row["price"])
if not price.is_finite() or price <= 0:
raise ValueError("price must be finite and positive")
except (ValueError, InvalidOperation) as exc:
quarantined.append({"id": row["id"], "raw": row.copy(),
"error": str(exc)})
else:
accepted.append({"id": row["id"], "value_usd": qty * price})
subtotal = sum((r["value_usd"] for r in accepted), Decimal("0"))
complete_valuation = False # No reconciliation or reject resolution here.
print(f"{len(accepted)} accepted, {len(quarantined)} quarantined")
print(f"Accepted subtotal: USD {subtotal}")
print(f"Complete valuation: {complete_valuation}")
print("Quarantined:", [(r["id"], r["raw"]["qty"]) for r in quarantined])
# 2 accepted, 1 quarantined
# Accepted subtotal: USD 350
# Complete valuation: False
# Quarantined: [('r2', 'bad')]Decimal from price text avoids an intermediate binary floating-point conversion. The arithmetic is exact for these small inputs; general decimal arithmetic still has context-dependent precision and rounding. Decimal parsing also accepts nonfinite values such as NaN, so successful parsing is not the whole validation rule. The quarantine retains the parser's error as well as the raw quantity and row ID; the input rows remain unchanged.Real position schemas would additionally check instrument identity, duplicate business keys, currency, snapshot timestamps, missing fields and bounds appropriate to the instrument. A non-positive price may be valid for a different data product; validation rules belong to a declared contract, not a universal market-data rule.
Retries must preserve meaning
Choose a stable batch identity and a business key before deciding whether a duplicate is an identical replay, a conflicting record or a legitimate revision. An upsert is one possible storage operation, not a complete duplicate policy: blindly overwriting a conflicting price can conceal a vendor correction. Bind retries to the same input snapshot and rule version. A new source revision gets an explicit revision identity and lineage rather than silently changing yesterday's result.
Publication needs its own failure boundary. Write a candidate output, check quality and completeness, then expose it through a transaction or a controlled version switch appropriate to the storage system. If publication fails after processing, a retry should reconcile the existing output before creating another release. A successful task status, or the fact that some rows loaded, does not itself certify a complete dataset.
Time and lineage matter to a quant
A vendor observation timestamp and the time the strategy could first obtain it answer different questions. A backtest that joins a late-arriving value to its earlier observation time can use future information. Preserve availability time, time zone, source revision and the join rule as part of the dataset's contract.
Sources
- Python
decimaldocumentation, especially decimal construction, context precision and finite-value checks. The example was executed with Python 3.14.4; the current documentation may describe a later release. - PostgreSQL 18
COPY, especially the distinction between default failure and best-effort loading withON_ERROR. Loader error policy is separate from a downstream consumer's completeness policy. - Apache Airflow, Best Practices: Creating a task, on retry outcomes, fixed input partitions and avoiding duplicate inserts. Airflow and PostgreSQL are source examples; neither is required or run by the Python snippet.