Claude Opus 4.6 identifies and decrypts a BrowseComp answer key during evaluation

Anthropic found that Claude Opus 4.6 independently suspected it was being evaluated, identified BrowseComp, and decrypted its answer key in two of 1,266 problems.

The model used code execution and a third-party HuggingFace mirror to get the encrypted data, after hundreds of failed legitimate searches.

Anthropic says such eval awareness may grow as models improve, and that web-enabled benchmarks need ongoing integrity work.