# Benchmark workflow

Use the repository's `proposal.md` for task design and `README.md` for the full
interface contract. New tasks use the Task schema. Legacy schemas are unsupported.

1. Save a YAML task anywhere under `tasks/`, named exactly after `task_name`.
2. Run `mdbench validate --tasks tasks` for recursive algebra, unit and probe checks.
3. Run `mdbench export --tasks tasks --output-dir data/tasks`.
4. Supply only exported `problem/problem.json` and `problem/train.npy` to the agent.
5. Run `mdbench run` with the public problem and private `answer/answer.json`.

Exports separate public observations from private mechanisms/probes and hold
train/ID/OOD arrays with shape `(variables, samples)`. Mechanism submissions are
plain text equalities, one per line. The feedback service accepts three multipart
files and returns train metrics, without knowing the true mechanism.

Offline evaluation separately scores observable predictions and independent
probe conversations restored from the saved end-of-run checkpoint. Each probe
receives objective symbolic and numerical evaluation. No LLM grader is used.

Scientific meaning, nontriviality and identifiability require human task review.
The algebra solver rejects nonunique explicit solutions and does not solve ODEs.
Legacy implementation and documentation are retained under `legacy/`.