The task
The agent writes a component that a finite element code calls during a simulation. In the first benchmarks this is a user element (UEL) for Abaqus/Standard, with the standard argument list: it returns the residual and the full stiffness matrix, keeps its own state variables, and can ask for a smaller time increment.
The model is from the published literature, and the task is not a test of new physics. The benchmarks test how AI agents implement a published model. They do not test or reproduce the source authors' results, and an agent's error is not an error in the source paper.
What the agent receives
- The model, restated as equations in a self-contained text, with its parameters.
- A public problem to practise on.
- A host program that calls the element through the standard Abaqus/Standard argument list.
- A one-line prompt that points to these files.
The agent works offline, inside its own folder, with no human help and a fixed time budget of about six to eight hours. Abaqus itself is not available to it.
What the agent decides
- How to integrate the model in time, including any substepping.
- How to compute the stiffness, and how to check it.
- The interpolation and quadrature of each field, and any extra variables the formulation needs.
- How and when history variables are stored and updated.
- How the element behaves on invalid states, and when it asks for a smaller increment.
How the work is judged
- Accuracy. The host's results with the agent's element are compared with an independent reference solution that is not shown to the agent, on the public problem and on hidden problems of the same kind.
- Robustness. Every problem completes with no NaN; history variables never decrease; Newton's method converges without avoidable cutbacks.
- The stiffness. Its accuracy, and how the agent obtains and checks it.
- The design. Whether the agent's choices are justified by evidence.
- Honesty. Whether the agent's claims match what its files show.