FrontierChallenge: Evaluating Scientific Workflow Completion

Aug 25, 2026·
Liangcai Su
,
Zhaopeng Feng
,
Zhuo Chen
,
Zhen Zhang
,
Xiang Lin
,
Ruilin Li
,
Handuo Zhang
,
Ning Wang
,
Kailong Wen
,
Yueqi Guo
,
Feng Xing
,
Yiling Guo
,
Brian Wang
,
Chenxiong Qian
,
Simon Shaolei Du
,
Lidong Bing
,
Xinyu Wang
· 3 min read
Type
Publication
arXiv preprint, arXiv:2608.24979
publication

Scientific agents can run analyses and write convincing reports while still failing to deliver the code, data, figures, and evidence a researcher needs. FrontierChallenge measures whether a specified scientific workflow is actually finished.

My contribution

My primary responsibility was data synthesis for FrontierChallenge. The benchmark design, evaluation, and results described below are the work of the broader team.

From raw inputs to a checkable handoff

The benchmark contains a pool of 300 workflows; the paper releases and evaluates 97 tasks across six scientific domains and 21 workflow families, with the other 203 retained internally. The released tasks span quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.

Each task supplies fixed inputs and an explicit set of required outputs. Task-specific graders check the submitted artifacts, including scientific calculations, structured files, figures, and reports. Pass Rate measures complete delivery; Avg. Score records partial progress. This distinction prevents a mostly correct analysis with missing or inconsistent outputs from being counted as a completed task.

Composition of the 97 released FrontierChallenge tasks across six domains and 21 workflow families.
The released evaluation set: domain coverage, workflow families, and hard/medium task composition. Source: FrontierChallenge, Figure 1.

View full-size task map.

What the evaluation revealed

The v2 paper evaluates twelve models using Codex, Claude Code, and FrontierAgent scaffolds. Its central result is a substantial gap between partial progress and complete delivery:

  • The highest Pass Rate is 20 of 97 tasks (20.6%), shared by GPT-5.6 Sol with Codex and Grok 4.6 with Claude Code. The highest Avg. Score reaches 87.9/100.
  • In analytical chemistry and electrochemistry/environment, the best partial scores reach 87.6 and 94.9, while the highest Pass Rates are only 4% and 0%, respectively.
  • Among 849 non-passing Claude Code runs, 641 (75.5%) end with language claiming completion. An agent’s final status message is therefore an unreliable substitute for checking its deliverables.

These are the paper’s reported results, not a live leaderboard. The task is to execute an already specified scientific workflow; it does not test whether an agent can independently choose a research agenda. The domain slices also have different sizes and should not be read as a ranking of disciplinary difficulty.

How the benchmark fits into agent evaluation

FrontierChallenge brings fixed scientific inputs, multiple interdependent artifacts, executable grading, and cross-domain coverage into one evaluation. The comparison below summarizes the paper’s positioning alongside related benchmarks; it is a comparison of design choices, not a performance ranking.

Comparison of scientific workflow coverage, fixed inputs, end-to-end execution, multiple artifacts, executable evaluation, and cross-domain science across related benchmarks.
Benchmark design comparison from FrontierChallenge, Table 1. Dots mark characteristics central to each benchmark’s design.

View full-size comparison.

The benchmark source and evaluation instructions are released within FrontierAgent, the agent runtime and research workbench I initiated and lead. The benchmark also connects to the process-based evaluation in Apodex Discovery: complete outputs and the quality of the investigation provide complementary evidence about an agent’s reliability.

Sources: Paper v2, 9 September 2026, especially Sections 3–4 and Table 2; arXiv record.

Agent Evaluation Scientific Workflows