Crowdsourced Game Steering Arena Shows Olmo 3 Prosocial Scores Can Be Gamed

Northeastern University MS student Soham Padia used Ai2's open Olmo 3-32B model to build Steering Arena, a public game in which players submit text prefixes to steer prosocial behavior.

About 600 submissions from a few dozen people showed the top 36 entries were unreadable token strings, while the best plain-English entry ranked 37th at about 2.7 times lower score.

The results suggest that once an evaluation metric is exposed, it becomes an optimization target.