FrontierChallenge: Evaluating Scientific Workflow Completion

Tue, 25 Aug 2026 00:00:00 +0000

Scientific agents can run analyses and write convincing reports while still failing to deliver the code, data, figures, and evidence a researcher needs. FrontierChallenge measures whether a specified scientific workflow is actually finished.

My contribution

My primary responsibility was data synthesis for FrontierChallenge. The benchmark design, evaluation, and results described below are the work of the broader team.

From raw inputs to a checkable handoff

The benchmark contains a pool of 300 workflows; the paper releases and evaluates 97 tasks across six scientific domains and 21 workflow families, with the other 203 retained internally. The released tasks span quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.

Each task supplies fixed inputs and an explicit set of required outputs. Task-specific graders check the submitted artifacts, including scientific calculations, structured files, figures, and reports. Pass Rate measures complete delivery; Avg. Score records partial progress. This distinction prevents a mostly correct analysis with missing or inconsistent outputs from being counted as a completed task.

The released evaluation set: domain coverage, workflow families, and hard/medium task composition. Source: FrontierChallenge, Figure 1.

What the evaluation revealed

The v2 paper evaluates twelve models using Codex, Claude Code, and FrontierAgent scaffolds. Its central result is a substantial gap between partial progress and complete delivery:

The highest Pass Rate is 20 of 97 tasks (20.6%), shared by GPT-5.6 Sol with Codex and Grok 4.6 with Claude Code. The highest Avg. Score reaches 87.9/100.
In analytical chemistry and electrochemistry/environment, the best partial scores reach 87.6 and 94.9, while the highest Pass Rates are only 4% and 0%, respectively.
Among 849 non-passing Claude Code runs, 641 (75.5%) end with language claiming completion. An agent’s final status message is therefore an unreliable substitute for checking its deliverables.

These are the paper’s reported results, not a live leaderboard. The task is to execute an already specified scientific workflow; it does not test whether an agent can independently choose a research agenda. The domain slices also have different sizes and should not be read as a ranking of disciplinary difficulty.

How the benchmark fits into agent evaluation

FrontierChallenge brings fixed scientific inputs, multiple interdependent artifacts, executable grading, and cross-domain coverage into one evaluation. The comparison below summarizes the paper’s positioning alongside related benchmarks; it is a comparison of design choices, not a performance ranking.

Benchmark design comparison from FrontierChallenge, Table 1. Dots mark characteristics central to each benchmark’s design.

The are released within , the agent runtime and research workbench I initiated and lead. The benchmark also connects to the process-based evaluation in : complete outputs and the quality of the investigation provide complementary evidence about an agent’s reliability.

Sources: , especially Sections 3–4 and Table 2; .

Scientific Workflows | Zhang Handuo's Site

FrontierChallenge: Evaluating Scientific Workflow Completion

My contribution

From raw inputs to a checkable handoff

What the evaluation revealed

How the benchmark fits into agent evaluation

Code and related work