FSDP Scaling

Key Insight

FSDP (Fully Sharded Data Parallel) shards a model's parameters, gradients, and optimizer state across all participating GPUs in the distributed group. To run a layer's computation, FSDP uses an AllGather collective operation to temporarily reconstruct the sharded weights, executes the forward or backward pass, and then immediately discards the full weights. By replacing DDP's large AllReduce with interleaved AllGather and ReduceScatter collectives, FSDP reduces per-GPU memory usage from O(1) to O(1/N) where N is the world size, allowing developers to scale training to massive model architectures.