Estimate a quantity by decomposing it into defensible factors, tracking units and testing assumptions, illustrated with a fictional quote archive.
Fermi estimation
Fermi estimation approximates an unfamiliar quantity by breaking it into simpler quantities that can be estimated and combined. In quant work, it helps establish the scale of a data archive, simulation or operational workload before detailed measurement is available. Its useful output is a number with an inspectable chain of assumptions.
A Fermi estimate need not come from a statistical sample. This distinguishes the method from point estimation under a sampling model. Some inputs may be measured; others may come from experience or a deliberately simplified scenario. State which is which.
How much quote data would we store?
Suppose a team wants an initial estimate of one day's uncompressed quote-record payload for a fictional research archive. It stipulates the following inputs; these are illustrative assumptions, not measurements of an exchange feed.
Factor
Assumption
What must eventually be checked
Symbol count N
100 symbols
Actual subscription universe
Average record rate q
2 records per second per symbol
Recorded events, including quiet periods and active symbols
Recording duration H
6.5 hours per day
Sessions and out-of-session coverage
Record size b
120 bytes per record
Actual encoded fields and record format
Define a record consistently: for example, one stored quote update. A feed message containing several updates is not automatically one record, and a snapshot of several price levels is not interchangeable with a single update. With that convention fixed, the decomposition is
Using decimal GB, where 1GB=109 bytes, the arithmetic gives 0.5616 GB per day. Report the rough estimate as about 0.6 GB per day of record payload. The extra digits document the calculation; they do not make the assumed quote rate accurate to four decimal places.
This target excludes indexes, replicas, protocol overhead and compression. Those change the stored or transmitted object. For example, doubling the retained record count and keeping three identical copies would multiply this payload by six, before any other effects. Each adjustment needs its own rationale; an unexplained “safety factor” hides the uncertainty instead of examining it.
Which assumption deserves the next measurement?
Hold the symbol count and duration fixed. If the rate is actually 4 rather than 2 records per second per symbol, payload doubles. If record size is also 240 rather than 120 bytes, it doubles again. The combined result is four times the base estimate, not twice.
Allowing q to range from 1 to 4 and b from 60 to 240 produces joint endpoint scenarios of 0.1404 and 2.2464 GB per day. They are one quarter and four times the base value, respectively. These are chosen sensitivity scenarios, not a confidence interval or guaranteed bounds on real storage.
Fictional quote-archive sensitivity: with 100 symbols and a 6.5-hour day, 2 records per second per symbol at 120 bytes gives 0.5616 decimal GB per day. Joint low and high scenarios give 0.1404 and 2.2464 GB per day.
Original sensitivity calculation for stipulated inputs. Line styles distinguish record sizes; the scenario endpoints have no assigned probabilities. Storage overhead and peak ingestion capacity are excluded.
For this product, a proportional change in any one factor produces the same proportional change in the result, holding the others fixed. What matters is how uncertain each factor is and how cheaply it can be checked. If symbol count and session duration are known but quote frequency is poorly grounded, sampling the actual recording stream is more informative than refining the hours-to-seconds conversion.
Do not assume estimation errors cancel. A busier market may increase both event frequency and record size, or the same mistaken definition of “record” may contaminate both inputs. Splitting a problem into many factors does not by itself establish independence or accuracy.
Use the estimate to design the check
A reusable workflow is to define the target and boundary, decompose it with units, attach a basis to each uncertain input, calculate its scale, and vary the doubtful inputs. Then challenge the result using a different route. Mahajan's treatment emphasises checking estimates with methods that draw on different background knowledge.
For the archive, compare the factor estimate with bytes written during a representative capture window, using the same payload definition. Include different activity periods before extrapolating to a full session. Rewriting the original multiplication in another order checks arithmetic, but supplies no new evidence about the rate assumption.
Finally, daily volume does not determine peak ingestion capacity. The same daily total can arrive smoothly or in bursts. A design for buffers, processing speed or network bandwidth needs measurements at the relevant short time scale. The Fermi estimate identifies what to investigate next; it cannot settle a different engineering question merely because the units look related.
Sources
Edward F. Redish, University of Maryland, Quantifying Experience: Estimation Problems, revised 7 September 2005: reasoning from defensible assumptions and reporting precision appropriate to the inputs.
Sanjoy Mahajan, The Art of Insight in Science and Engineering: Mastering Complexity, MIT Press, 2014, §1.1 and §1.5: decomposition into simpler estimates and checks using different methods. The quote-archive example and sensitivity figure here are original stipulated scenarios.
