Optimize a single reaction
Task spaceMaximize the yield of an amide-coupling reaction by testing conditions across three successive experimental plates in a simulated laboratory.
Finding effective reaction conditions requires balancing exploration with refinement of promising results. A limited experimental budget makes the choice of each plate important: agents must use earlier measurements to decide what to test next.
Agents design and execute 24 experiments in three batches of eight, review the measured yields after each plate, and nominate one tested condition. Scoring uses the mean yield of that condition in separate confirmation experiments, testing whether the search identifies a condition that performs reliably beyond the initial measurements.
Scoring
The confirmation yield must close at least 90% of the gap from the random-search baseline to the task-constrained oracle. This is a normalized improvement threshold, not 90% raw yield.
RESULTS
Maximize the yield of an amide-coupling reaction by testing conditions across three successive experimental plates in a simulated laboratory.
| Model | Pass@1 | Cost ($)Model/API inference cost only, averaged per task attempt. Lab and labor costs are not reported. Repeats are averaged using the same task and family weights as scores. | |
|---|---|---|---|
 Gemini 3.8 Flashhigh | 26.6% | $0.552 | |
 Claude Opus 5xhigh | 20.3% | $1.47 | |
 Grok 4.6xhigh | 14.1% | $0.874 | |
| Claude Fable 5.1xhigh | 7.8% | $2.34 | |
 GPT–5.6 Solxhigh | 4.7% | $1.18 | |
| GPT–6 Astraxhigh | 4.7% | $3.32 | |
0%50%100% | |||
Whiskers show ±1 standard error from repeats.
Model trajectories
One example per model.
Run grading
Pass
| Measurement | Recorded result | Outcome |
|---|---|---|
| Normalized score | 100.0 % | ✓ Pass |
Pass condition: The confirmation yield must close at least 90% of the gap from the random-search baseline to the task-constrained oracle. This is a normalized improvement threshold, not 90% raw yield.
Optimize a single reactionGemini 3.8 Flash high
Gemini 3.8 Flash highRun gradingPass
00:0000:01
Model activity
Loading transcript…