Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

Tue, 11 Aug 2026 00:00:00 +0000

Real investigations rarely begin with a fully specified question, a clean dataset, and an immediately available answer key. Apodex Discovery studies the whole solver system—model, execution harness, tools, and control policies—and how it can pursue an extended, verifiable investigation.

My contribution

My work focused on evaluating agents through established harnesses and scoring their execution trajectories:

Ran benchmark tasks with mainstream harnesses to assess Apodex 1.1, Apodex 1.1 Mini, and other leading models.
Applied HDS6 multidimensional rubrics at the trajectory level to diagnose how agents used tools, handled evidence, maintained coherence, and responded to failure, complementing final outcome scores.
Helped separate model assessment from dependence on our in-house harness, and contributed difficult problems in domains where I have expertise through discussions with the team.

The verification–repair examples below illustrate the team’s reported findings and the value of trajectory-level evaluation.

Turning a research problem into an executable task

The framework connects three pieces: a problem manifest that defines objectives and constraints, an environment that exposes data and tools, and verification that checks both outcomes and the investigation that produced them. Its TRACES benchmark covers 17 executable environments and 218 episodes in the technical report, spanning biomedicine, clinical translation, and LLM engineering.

A research mission combines a model with an execution environment and explicit verification. Process feedback can guide correction without revealing the hidden outcome.

HDS6: inspecting the process, not just the answer

HDS6 assesses six capabilities: Tools, Repair, Alternatives, Coherence, Evidence, and Scope. A process verifier identifies concrete deficiencies and cites the trajectory steps that support its judgment. This makes a low score actionable: the solver can see which assumption, tool use, or missing piece of evidence needs attention.

For verification-driven repair, those findings are condensed into a repair note and supplied to a new run of the same task. The note excludes the hidden ground truth and outcome score. The two cases below illustrate how this changes the work, beyond making the final report sound more confident.

Case study: a correct table that could not be audited

In a clinical safety-reporting episode, the solver produced the correct numerical table but chose an under-specified patient population before declaring its assumption. Its evidence ledger also omitted the patient IDs behind the denominator, preventing an independent recount.

The process critique targeted those two gaps. In the repaired run, the solver clarified its population choice with the environment’s virtual clinician and recorded IDs for both numerator and denominator. The table values stayed the same; evidence and traceability improved from 0.71 to 1.00. The distinction matters: a correct number and a reproducible, accountable analysis are different deliverables.

Clinical reporting repair: make assumptions explicit before computation and preserve the evidence needed to recount each cell. See report Section 4.4.1. The illustration’s 0.83 → 0.95 is termed an outcome score in the report, although the illustration labels it a process score.

· .

Case study: an estimator named but never executed

In a pretraining-data scouting episode, the solver claimed to use capture–recapture estimation but assumed each source held about 100 documents. The process critique identified the missing measurement and directed it to use document fingerprints it had already collected.

The repaired run estimated source sizes from repeated observations. Its unique-document estimate moved from 108 to 2,471, against a hidden reference of 1,505, and the estimation score rose from 0.05 to 0.30. This was a measurable improvement, not a solved problem: the estimate still overshot the reference and required additional sampling budget.

Corpus scouting: the critique exposes an unsupported input to the estimator. Measuring that input improves the result, but residual calibration error remains. Source: report Section 4.4.2.

· .

Reported results and what they establish

Across 434 initially deficient trajectories with HDS6 below 3, one repair rerun raised the environment-specific outcome score by 0.155 on average. Of these runs, 204 improved, 144 were unchanged, and 86 declined. The report explicitly notes that there was no matched rerun-without-feedback control, so this experiment cannot fully separate the effect of repair guidance from ordinary run-to-run variation.

The useful result is a concrete evaluation-and-repair procedure: locate a weakness in the trajectory, turn it into feedback without exposing the answer, and measure what changes in the subsequent run.

Sources: , Sections 3 and 4.4, Table 13, and Figures 8–9. Visit the for current environments and results. Related infrastructure: .