The 37 Details

Key Insight

PPO's reputation for "just working" comes less from its five-line clipped objective than from roughly three dozen small implementation choices layered on top — things like advantage normalization, clipping the value function loss, orthogonal weight initialization, annealing the learning rate to zero, reward scaling, and global gradient clipping. Individually each looks like a minor detail; together they are the difference between a PPO that matches published scores and one that quietly fails to learn. This project implements or audits every one of the 37 documented details and measures each as its own ablation, so you learn not just that they matter but how much each contributes. The deeper lesson generalizes beyond PPO: in modern RL the gap between a paper's pseudocode and a working agent is paved with unglamorous engineering.