FrontierChallenge: Evaluating Scientific Workflow Completion

Tue, 25 Aug 2026 00:00:00 +0000

Scientific agents can run analyses and write convincing reports while still failing to deliver the code, data, figures, and evidence a researcher needs. FrontierChallenge measures whether a specified scientific workflow is actually finished.

My contribution

My primary responsibility was data synthesis for FrontierChallenge. The benchmark design, evaluation, and results described below are the work of the broader team.

From raw inputs to a checkable handoff

The benchmark contains a pool of 300 workflows; the paper releases and evaluates 97 tasks across six scientific domains and 21 workflow families, with the other 203 retained internally. The released tasks span quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.

Each task supplies fixed inputs and an explicit set of required outputs. Task-specific graders check the submitted artifacts, including scientific calculations, structured files, figures, and reports. Pass Rate measures complete delivery; Avg. Score records partial progress. This distinction prevents a mostly correct analysis with missing or inconsistent outputs from being counted as a completed task.

The released evaluation set: domain coverage, workflow families, and hard/medium task composition. Source: FrontierChallenge, Figure 1.

What the evaluation revealed

The v2 paper evaluates twelve models using Codex, Claude Code, and FrontierAgent scaffolds. Its central result is a substantial gap between partial progress and complete delivery:

The highest Pass Rate is 20 of 97 tasks (20.6%), shared by GPT-5.6 Sol with Codex and Grok 4.6 with Claude Code. The highest Avg. Score reaches 87.9/100.
In analytical chemistry and electrochemistry/environment, the best partial scores reach 87.6 and 94.9, while the highest Pass Rates are only 4% and 0%, respectively.
Among 849 non-passing Claude Code runs, 641 (75.5%) end with language claiming completion. An agent’s final status message is therefore an unreliable substitute for checking its deliverables.

These are the paper’s reported results, not a live leaderboard. The task is to execute an already specified scientific workflow; it does not test whether an agent can independently choose a research agenda. The domain slices also have different sizes and should not be read as a ranking of disciplinary difficulty.

How the benchmark fits into agent evaluation

FrontierChallenge brings fixed scientific inputs, multiple interdependent artifacts, executable grading, and cross-domain coverage into one evaluation. The comparison below summarizes the paper’s positioning alongside related benchmarks; it is a comparison of design choices, not a performance ranking.

Benchmark design comparison from FrontierChallenge, Table 1. Dots mark characteristics central to each benchmark’s design.

The are released within , the agent runtime and research workbench I initiated and lead. The benchmark also connects to the process-based evaluation in : complete outputs and the quality of the investigation provide complementary evidence about an agent’s reliability.

Sources: , especially Sections 3–4 and Table 2; .

Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

Tue, 11 Aug 2026 00:00:00 +0000

Real investigations rarely begin with a fully specified question, a clean dataset, and an immediately available answer key. Apodex Discovery studies the whole solver system—model, execution harness, tools, and control policies—and how it can pursue an extended, verifiable investigation.

My contribution

My work focused on evaluating agents through established harnesses and scoring their execution trajectories:

Ran benchmark tasks with mainstream harnesses to assess Apodex 1.1, Apodex 1.1 Mini, and other leading models.
Applied HDS6 multidimensional rubrics at the trajectory level to diagnose how agents used tools, handled evidence, maintained coherence, and responded to failure, complementing final outcome scores.
Helped separate model assessment from dependence on our in-house harness, and contributed difficult problems in domains where I have expertise through discussions with the team.

The verification–repair examples below illustrate the team’s reported findings and the value of trajectory-level evaluation.

Turning a research problem into an executable task

The framework connects three pieces: a problem manifest that defines objectives and constraints, an environment that exposes data and tools, and verification that checks both outcomes and the investigation that produced them. Its TRACES benchmark covers 17 executable environments and 218 episodes in the technical report, spanning biomedicine, clinical translation, and LLM engineering.

A research mission combines a model with an execution environment and explicit verification. Process feedback can guide correction without revealing the hidden outcome.

HDS6: inspecting the process, not just the answer

HDS6 assesses six capabilities: Tools, Repair, Alternatives, Coherence, Evidence, and Scope. A process verifier identifies concrete deficiencies and cites the trajectory steps that support its judgment. This makes a low score actionable: the solver can see which assumption, tool use, or missing piece of evidence needs attention.

For verification-driven repair, those findings are condensed into a repair note and supplied to a new run of the same task. The note excludes the hidden ground truth and outcome score. The two cases below illustrate how this changes the work, beyond making the final report sound more confident.

Case study: a correct table that could not be audited

In a clinical safety-reporting episode, the solver produced the correct numerical table but chose an under-specified patient population before declaring its assumption. Its evidence ledger also omitted the patient IDs behind the denominator, preventing an independent recount.

The process critique targeted those two gaps. In the repaired run, the solver clarified its population choice with the environment’s virtual clinician and recorded IDs for both numerator and denominator. The table values stayed the same; evidence and traceability improved from 0.71 to 1.00. The distinction matters: a correct number and a reproducible, accountable analysis are different deliverables.

Clinical reporting repair: make assumptions explicit before computation and preserve the evidence needed to recount each cell. See report Section 4.4.1. The illustration’s 0.83 → 0.95 is termed an outcome score in the report, although the illustration labels it a process score.

· .

Case study: an estimator named but never executed

In a pretraining-data scouting episode, the solver claimed to use capture–recapture estimation but assumed each source held about 100 documents. The process critique identified the missing measurement and directed it to use document fingerprints it had already collected.

The repaired run estimated source sizes from repeated observations. Its unique-document estimate moved from 108 to 2,471, against a hidden reference of 1,505, and the estimation score rose from 0.05 to 0.30. This was a measurable improvement, not a solved problem: the estimate still overshot the reference and required additional sampling budget.

Corpus scouting: the critique exposes an unsupported input to the estimator. Measuring that input improves the result, but residual calibration error remains. Source: report Section 4.4.2.

· .

Reported results and what they establish

Across 434 initially deficient trajectories with HDS6 below 3, one repair rerun raised the environment-specific outcome score by 0.155 on average. Of these runs, 204 improved, 144 were unchanged, and 86 declined. The report explicitly notes that there was no matched rerun-without-feedback control, so this experiment cannot fully separate the effect of repair guidance from ordinary run-to-run variation.

The useful result is a concrete evaluation-and-repair procedure: locate a weakness in the trajectory, turn it into feedback without exposing the answer, and measure what changes in the subsequent run.

Sources: , Sections 3 and 4.4, Table 13, and Figures 8–9. Visit the for current environments and results. Related infrastructure: .

Agent Evaluation | Zhang Handuo's Site

FrontierChallenge: Evaluating Scientific Workflow Completion

My contribution

From raw inputs to a checkable handoff

What the evaluation revealed

How the benchmark fits into agent evaluation

Code and related work

Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

My contribution

Turning a research problem into an executable task

HDS6: inspecting the process, not just the answer

Case study: a correct table that could not be audited

Case study: an estimator named but never executed

Reported results and what they establish