Data compression

Data compression encodes data so it can usually be stored or transmitted in fewer bits. A quant team can fit more market data into an archive or move a replay file faster when bandwidth is the bottleneck. Lossless compression recovers the original bytes exactly. Lossy compression permits an approximation. Rounding quote prices or timestamps before storage can merge distinct prices or event times, changing what a replay can reconstruct. A lossless codec applied after rounding preserves only the rounded data.

Take the toy byte sequence AAAABBCCC. A run is a stretch of consecutive copies of the same byte. This run-length encoder writes two bytes per run: the symbol and an unsigned count from 1 to 255. It splits longer runs into multiple pairs. Here it writes (A, 4), (B, 2) and (C, 3). The parentheses and commas just display the pairs, and 4 denotes a count byte with value four. The decoder reads each pair in order and repeats its symbol the indicated number of times, recovering AAAABBCCC byte for byte.

Using original bytes divided by compressed bytes, the payload compression ratio is $9/6=1.5$, or 1.5 to 1. The fraction of original bytes saved is $(9-6)/9=1/3$. Both sizes exclude framing. Three extra bytes of framing on the compressed side would erase the saving if the raw representation needed no corresponding overhead. With ABABABABA, every byte starts a new run. The same codec produces 18 payload bytes and a ratio of $9/18=0.5$, so the payload doubles. Lossless compression cannot shorten every possible input. RFC 1951, §1.1

Data serialization decides how structured records are represented as bytes. Compression can exploit patterns within that representation. For example, Apache Parquet can store repeated column values in a dictionary and encode the resulting IDs with a hybrid of run-length encoding and bit packing. For integer data it also supports delta encoding, which records a starting value and encodes successive differences. If timestamp increments are close together, offsets from the smallest increment in a block can take fewer bits. Parquet can then apply a codec such as Zstandard to page data. The page has a separate header, so a page-body measurement and a complete file measurement answer different questions. Apache Parquet encodings, page compression

Run-length encoding depends on adjacency. A frequency-based method can help even when frequent symbols are interleaved. Huffman coding assigns shorter codes to more frequent symbols where the frequency distribution permits it. The SelfMadeQuant lesson builds the code tree from frequencies. Its preview is public and the full lesson requires membership. Huffman’s original paper, Greedy Algorithms and Correctness Proofs

Suppose two complete files contain the same replay data and take 900 MB uncompressed versus 600 MB compressed, including file metadata. The storage saving is 300 MB. At a sustained 1 Gbit/s, transferring the file bytes alone takes 7.2 seconds versus 4.8 seconds, using decimal units. If compression and decompression run in series with the transfer and all other costs are equal, they have 2.4 seconds combined before they erase the transfer advantage. For live feeds, batch buffering and decode time also affect when a message becomes usable. Compare end-to-end market data latency under the intended workload alongside bytes saved.