The Zero Redundancy Optimiser (ZeRO) is a family of distributed-training memory techniques that partitions model state across data-parallel processes instead of storing every copy on every device. For a quant training a large forecasting or language model, it can make the training state fit within the available GPU memory. It changes state placement rather than replacing the underlying optimiser's update rule.
Three stages of removing duplication
Ordinary replicated data parallelism gives each process different training examples but a full copy of the model state. ZeRO progressively shares responsibility for storing and updating that state:
| Stage | Partitioned across processes | Still replicated in the ideal model-state accounting |
|---|---|---|
| 1 | Optimiser state | Model weights and gradients |
| 2 | Optimiser state and reduced gradients | Model weights |
| 3 | Optimiser state, reduced gradients and model weights | None of these three state groups |
The stages are cumulative. Here, optimiser state includes a high-precision master weight copy when the precision recipe uses one, as well as the optimiser's history arrays. Adam, for example, keeps first- and second-moment estimates.
A gradient partition is not just the gradient from its owner's local mini-batch. Contributions from the data-parallel processes must be reduced so that the owner can apply the intended update. At stage 3, missing weights are gathered when needed for forward and backward computation, then can be released. Sharded ownership therefore does not mean a process computes forever using only its stored weights. The official DeepSpeed tutorial describes the cumulative stages; the original paper's Sections 5.1–5.3 explain their communication.
A twelve-parameter memory ledger
Use the paper's specific FP16/FP32 mixed-precision training convention. Each scalar parameter requires:
- 2 bytes for its FP16 model weight;
- 2 bytes for its FP16 gradient;
- 12 bytes for three FP32 values: a master weight, first moment and second moment.
The last group is counted as optimiser state. It is not 12 bytes for the two moments alone. Section 3.1 of the paper gives this convention; other precision and optimiser choices have different byte budgets.
Let be the number of scalar parameters and the data-parallel degree. Assume equal partitions, all parameters trainable and no offloading. The ideal retained model-state bytes per device are
For the fictional , example, each sharded group assigns four parameters' worth of state to a device:
| Arrangement | Weights | Gradients | Master weights + moments | Total per device |
|---|---|---|---|---|
| Replicated | 24 bytes | 24 bytes | 144 bytes | 192 bytes |
| ZeRO stage 1 | 24 bytes | 24 bytes | 48 bytes | 96 bytes |
| ZeRO stage 2 | 24 bytes | 8 bytes | 48 bytes | 80 bytes |
| ZeRO stage 3 | 8 bytes | 8 bytes | 48 bytes | 64 bytes |
Original model-state accounting example, not a measured GPU benchmark. Stages progressively remove duplicated storage; transient working memory is excluded.
Stage 1 halves the retained state in this example; it does not divide the entire 192-byte total by three. Only stage 3 partitions all three groups. With one device, none of these partitions saves storage.
Why 64 bytes is not a peak-memory prediction
The ledger excludes activations, temporary parameter gathers, communication buffers, padding and allocator overhead. Those exclusions matter even when every permanent state group is sharded. In particular, DeepSpeed documents that each submodule's forward and backward work must fit in device memory; a full layer can still be too large. Its ZeRO documentation, checked on 1 November 2026, describes this constraint and tiling support.
Partitioning and offloading are separate choices. Sharding assigns state to different processes; offloading moves state to CPU or NVMe storage. The current documentation exposes offloading separately from the stage setting. Neither the ideal byte formula nor a higher stage establishes faster training: communication and the workload still matter.
For the underlying history arrays, the optimiser-choices lesson develops Adam's two accumulator recurrences. Its preview is public; the full lesson requires membership.
Sources
- Rajbhandari, S., Rasley, J., Ruwase, O. and He, Y. (2020), ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, v3; Sections 2.2.3, 3.1–3.2 and 5.1–5.3. The first submission was in 2019; v3 was revised on 13 May 2020.
- DeepSpeed, Zero Redundancy Optimizer tutorial, overview, stage-1/stage-2 examples and ZeRO-Infinity discussion; checked 1 November 2026.
- DeepSpeed, ZeRO documentation, stage definitions, configuration, offloading and assumptions; documentation identified itself as version 0.19.9 when checked on 1 November 2026.