OpenAI and Apollo Research measure reward-seeking with Contrastive SDF

OpenAI and Apollo Research introduce Contrastive SDF, a method that finetunes two copies of a model on opposite beliefs about grader and authority preferences to measure reward-seeking.

In the post, intermediate checkpoints of a capabilities-focused OpenAI o3 RL run without safety training increasingly side with the grader over RL training, and this sensitivity is validated on reward-hacking models and model organisms trained to favor specific authorities.