Memory allocators and memory pools

A memory allocator hands out blocks of dynamic storage and manages them when they are returned. A memory pool keeps backing storage for a class of allocations so it can serve many requests from that storage. In a low-latency trading system, a burst of new price levels can turn a cheap allocation repeated once per event into substantial work. A pool can reuse nodes when levels disappear, but its capacity still has to cover the number of live nodes during a burst. Operating Systems: Three Easy Pieces, Boost.Pool

Suppose a toy allocator uses slots of 32, 64 and 128 bytes. It places each request in the smallest slot that can hold it. Requests of 24, 48 and 80 bytes occupy slots of 32, 64 and 128 bytes respectively. That is 152 bytes requested across 224 bytes of slots. The difference is 72 bytes of internal fragmentation, made up of 8, 16 and 48 bytes of slack inside the occupied slots. These are fictional size classes. The calculation excludes metadata, unused slots and page rounding. An actual allocator's class table depends on its implementation and configuration, and a request can also impose memory alignment constraints. jemalloc manual

External fragmentation concerns free space between live blocks. In a separate fixed region, suppose a live 24-byte block sits between two free 40-byte blocks. There are 80 free bytes in total, but a request for a single contiguous 64-byte block cannot fit in that region. Merging adjacent free blocks can help when the intervening block is released. An allocator that can grow may satisfy the request elsewhere. Operating Systems: Three Easy Pieces

A fixed-size pool can put unused slots on a free list. To allocate a price-level node, it takes a slot from the list. When the level is removed and the node is no longer in use, it puts the slot back. If the list empties, a growable pool needs more backing storage. A bounded pool needs a policy for exhaustion, such as falling back to another allocator or failing the request. A warm run that never exhausts the pool can miss the cost of acquiring more storage. Boost.Pool

A batch of scenario-pricing scratch objects has a different lifetime. If all of them expire together, a bump-pointer arena can advance through backing blocks as it allocates and make the storage reusable in one reset. Any object or reference still needed after that reset must live elsewhere. LLVM's bump-pointer allocator

An allocation count measures how often requests occur. Live heap bytes measure how much dynamic storage is in use at a point in time. Allocators may keep freed space available for reuse, and a partially occupied page can still back live allocations. Process resident set size (RSS) measures memory currently resident in RAM across the process's mappings, including resident file-backed and shared mappings on Linux. RSS can stay high after live allocations fall. To see where the bytes are, compare the allocator's own counters with the operating system's mapping-level breakdown. In jemalloc, for example, stats.allocated counts allocated bytes, stats.active measures the bytes in pages backing active allocations, and stats.resident estimates an upper bound on allocator-resident bytes. jemalloc manual, Linux kernel proc documentation

An allocator comparison in an order-book replay depends on request sizes, object lifetimes, concurrency and burst patterns. Measure allocation counts, memory and event tail latency under the same replay. Include startup or pool growth when those phases can occur in production. Thread-local caches in a general-purpose allocator can make reuse cheap already, while retaining more memory. The SelfMadeQuant Queue-Position and Latency-Aware Order-Book Simulator report shows a free-list allocator for price-level nodes and a separate allocation-count check that caught an injected per-event allocation. The report requires membership. jemalloc manual