Multi-GPU DDP

Key Insight

Training models with DDP (Distributed Data Parallel) replicates the model across multiple GPUs, where each GPU processes a different slice of the training batch in parallel. During the backward pass, the GPUs perform an AllReduce collective operation to synchronize and sum their gradients before updating weights. Profiling the execution shows that while computation scales linearly with the number of GPUs, communication overhead from gradient synchronization can bottleneck training if network bandwidth is insufficient.