Tri Dao highlights SonicMoE's mathematical backward rewrite for faster MoE on Blackwell
Original title@WentaoGuo7 explains really well this nice mathematical rewrite of the MoE backward. This leads to much lower activation mem, and way fas...
AISummary
Tri Dao points to Wentao Guo's explanation of a mathematical rewrite of the MoE backward pass, which reduces activation memory and speeds up training, especially for fine-grained MoE. He also notes the work leverages new Blackwell features, including 2CTA MMA and CLC, to build fast MoE kernels.
Source: Tri Dao · x.comPublished · added here