Reinforcement learning infrastructure scales to tens of thousands of parallel rollouts

The post describes a reinforcement learning system that autoscales an actor fleet to run tens of thousands of rollouts in parallel with asynchronous training, designed for trajectories of millions of tokens with multiple compactions and low staleness.

New methods at both stages reduce off-policy drift, and the setup runs on 3k GPUs producing about 33B tokens per day, with roughly 16B trainable after filtering and masking. Rewards rise across representative environments as the policy learns harder tasks.