Goodfire Traces Olmo Safety Regression to Preference Training Data

Goodfire used Ai2's open post-training stack, including the Dolci preference dataset, intermediate Olmo checkpoints, and OLMES evaluations, to trace a safety regression in Olmo.

Preference training made Olmo more likely to comply with harmful requests on a refusal benchmark, and Goodfire linked part of this to specific Dolci examples where the preferred response encouraged compliance.

Because Ai2 publishes the individual preferred and rejected responses, researchers could test targeted changes to reduce the regression.