MechanicsHarness

How a benchmark is built

The procedure below follows the records of the rock benchmark. Everything that judges an agent is fixed and tested before any agent runs.

  1. Choose a published model and a test. Credit the reference work exactly: the paper and any code from its authors that we used, what we took from it, what is ours, and what we left out.
  2. Fix the test conditions from the source. Geometry, loading, boundary conditions and parameters are checked against the source. The paper is the source of truth, and our code is checked against it.
  3. Restate the model as a self-contained task. The text fixes the model, the parameters, the element contract and a public problem, and leaves the numerical method to the agent.
  4. Provide a host. A model-free program compiles the agent's element, calls it through the standard Abaqus/Standard argument list, and solves each increment by Newton's method on the element's residual and stiffness, cutting the increment back when needed, as Abaqus/Standard does.
  5. Build an independent reference before any agent runs. Where two references are built separately, they must agree before a problem is used.
  6. Write hidden problems of the same kind as the public one. They never enter the agent's workspace.
  7. Set the tolerances before any agent runs. They come from the spread between the references and the time-step error at the public step, not from agent results.
  8. Test the test. A known-good element and several known-wrong elements go through the host and the scorer first. Each known-wrong element must fail.
  9. Freeze and run. The agent's starting folder is frozen by file hashes. Each agent runs alone in a sandbox with a fresh home folder, no internet, a one-line prompt and a time limit.
  10. Score and review. The scorer reports accuracy, robustness and the solver's behaviour, and edge cases not told to the agents probe further. A person asks what each mismatch means before calling it a failure, and separates defects of the agent from defects of our host or scorer.
  11. Record and share. The session records, the agents' folders and the grades are kept. Results go to the authors of the source paper before publication, and their corrections are recorded.