Hidden tests
Tests reserved for future blind evaluation, and their reference results, stay private. We publish the properties they test and this policy. Completed historical tests may be released as evaluation cases; once released, they are no longer hidden, and later rounds use new private cases.
Traceable verdicts
Every published verdict will identify the task, the agent and model version, the tools, the resource budget, the submission, the host and scorer versions, and the reference data. Where a fix to the host or the scorer affects a result, the saved submissions are rescored on a common corrected evaluator, the original results are kept, and evaluation changes made after the runs are labelled.
Agent errors and evaluator errors
Case studies distinguish defects of the agent, defects of our own host or scorer, and outcomes that remain unresolved. The runs form a small collection and are described as case studies; we avoid broad rankings across different task conditions.
Human verification
Each case study states who specified the physics, who interpreted the source paper, who wrote or checked the reference, who reviewed discrepancies, and who made each final judgment.
Source papers and their authors
Results are shared with the authors of the source paper before publication, and any factual corrections they provide are recorded. Source papers are cited and linked. Third-party material is included only where its redistribution terms permit, and files that authors share with us privately stay private.
Agent records
Session records are kept for every run. Transcripts or excerpts are released only after review for credentials, personal information and private correspondence.