Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence
Real investigations rarely begin with a fully specified question, a clean dataset, and an immediately available answer key. Apodex Discovery studies the whole solver system—model, execution harness, tools, and control policies—and how it can pursue an extended, verifiable investigation.
My contribution
My work focused on evaluating agents through established harnesses and scoring their execution trajectories:
- Ran benchmark tasks with mainstream harnesses to assess Apodex 1.1, Apodex 1.1 Mini, and other leading models.
- Applied HDS6 multidimensional rubrics at the trajectory level to diagnose how agents used tools, handled evidence, maintained coherence, and responded to failure, complementing final outcome scores.
- Helped separate model assessment from dependence on our in-house harness, and contributed difficult problems in domains where I have expertise through discussions with the team.
The verification–repair examples below illustrate the team’s reported findings and the value of trajectory-level evaluation.
Turning a research problem into an executable task
The framework connects three pieces: a problem manifest that defines objectives and constraints, an environment that exposes data and tools, and verification that checks both outcomes and the investigation that produced them. Its TRACES benchmark covers 17 executable environments and 218 episodes in the technical report, spanning biomedicine, clinical translation, and LLM engineering.
HDS6: inspecting the process, not just the answer
HDS6 assesses six capabilities: Tools, Repair, Alternatives, Coherence, Evidence, and Scope. A process verifier identifies concrete deficiencies and cites the trajectory steps that support its judgment. This makes a low score actionable: the solver can see which assumption, tool use, or missing piece of evidence needs attention.
For verification-driven repair, those findings are condensed into a repair note and supplied to a new run of the same task. The note excludes the hidden ground truth and outcome score. The two cases below illustrate how this changes the work, beyond making the final report sound more confident.
Case study: a correct table that could not be audited
In a clinical safety-reporting episode, the solver produced the correct numerical table but chose an under-specified patient population before declaring its assumption. Its evidence ledger also omitted the patient IDs behind the denominator, preventing an independent recount.
The process critique targeted those two gaps. In the repaired run, the solver clarified its population choice with the environment’s virtual clinician and recorded IDs for both numerator and denominator. The table values stayed the same; evidence and traceability improved from 0.71 to 1.00. The distinction matters: a correct number and a reproducible, accountable analysis are different deliverables.
View full-size illustration · Download vector PDF.
Case study: an estimator named but never executed
In a pretraining-data scouting episode, the solver claimed to use capture–recapture estimation but assumed each source held about 100 documents. The process critique identified the missing measurement and directed it to use document fingerprints it had already collected.
The repaired run estimated source sizes from repeated observations. Its unique-document estimate moved from 108 to 2,471, against a hidden reference of 1,505, and the estimation score rose from 0.05 to 0.30. This was a measurable improvement, not a solved problem: the estimate still overshot the reference and required additional sampling budget.
View full-size illustration · Download vector PDF.
Reported results and what they establish
Across 434 initially deficient trajectories with HDS6 below 3, one repair rerun raised the environment-specific outcome score by 0.155 on average. Of these runs, 204 improved, 144 were unchanged, and 86 declined. The report explicitly notes that there was no matched rerun-without-feedback control, so this experiment cannot fully separate the effect of repair guidance from ordinary run-to-run variation.
The useful result is a concrete evaluation-and-repair procedure: locate a weakness in the trajectory, turn it into feedback without exposing the answer, and measure what changes in the subsequent run.
Sources: Full technical report, Sections 3 and 4.4, Table 13, and Figures 8–9. Visit the TRACES website for current environments and results. Related infrastructure: Prax / AgentOS and ApodexHarness.