An Executable-Gate Multi-Agent Research Organization: Artifact, Case Study, and a Pre-Registered Gate-Calibration Study

Overview

The artifact is a complete research organization expressed as 39 Claude agent role files plus an orchestrator that runs as the main session, in 13 divisions — leadership, frontier research, theory and statistics, interpretability, research engineering, evaluation and red team, safety and governance, literature, scientific writing, visualization, program management, publication, and a self-improvement loop — operating over a 9-phase lifecycle with 4 human checkpoints.

The structural rule that defines the system: nothing self-certifies. Every role output passes an executable gate (tests, make verify, citation checks, grep gates) plus adversarial sign-off from a paired critic role holding kill authority.

The Two Empirical Records

Record 1 — the v0.1.0 run. The organization examined five candidate research directions and abandoned all five at the gates. Its flagship integrity exhibit was retracted after failing its own verifier. Zero theses survived. This is reported as a finding, not buried: a single run (n = 1) consistent with either working quality control or uniform rejection.

Record 2 — the calibration study. A pre-registered (git tag calibration-prereg-v1), blinded, seeded study distinguished the two hypotheses. Twenty theses — ten with planted defects across six flaw classes, ten known-sound reproductions — were evaluated under salted filenames with SHA-256 sealed labels, in two arms: M1-ON (round-1 auto-fail prior active) and M1-OFF (ablation).

Headline Results

Metric	M1-ON	M1-OFF (ablation)
Sensitivity (correct-reason attribution)	13/15 = 0.87 [0.60, 0.98]	14/15 = 0.93 [0.68, 1.00]
Plants missed entirely	0 of 15	0 of 15
Known-flawed theses killed	10 of 10	10 of 10
Pooled validity-gate specificity (exploratory, L8)	36/50 = 0.72 [0.58, 0.84]	48/50 = 0.96 [0.86, 1.00]
Known-sound theses killed (exploratory, L8)	8 of 10	1 of 10

Intervals are 95% exact Clopper–Pearson. What stands: attribution is a within-arm measurement against sealed per-plant ground truth — the correct flaw class at the correct location for 13 of 15 plants (M1-ON) and 14 of 15 (M1-OFF). No plant was missed entirely in either arm, which is the same fact as every known-flawed thesis being killed; that is raw thesis-level rejection, which a uniform rejector or a genre heuristic would also produce, so it is reported as measured rather than as unaffected. The gates are not a uniformly rejecting filter: even allowing for the genre confound, sound theses were not uniformly killed (specificity far above the near-zero a uniform rejector predicts, most clearly in M1-OFF, where 1 of 10 was killed). The rows marked exploratory compare the two corpus arms and were downgraded by Erratum v0.2.1 (below). As originally reported, the round-1 auto-fail prior — "nothing passes review on the first try," a plausible-sounding skeptical default — lowered pooled specificity from 0.96 to 0.72 and killed 8 of 10 known-sound theses, with no observed sensitivity benefit (0.87 vs 0.93; the intervals overlap). The default was flipped to off in v0.2.0 and stays off as a design choice; its rationale is now exploratory, and the new default is unvalidated on fresh data (the calibration corpus is spent).

Erratum v0.2.1 (June 13, 2026)

The erratum is documentation-only: no code, configuration or calibration data changed. An independent F6 second review (a separate same-model-family LLM reviewer) on 2026-06-12, carried out while the successor program (C1) was selecting its problem, found a genre confound between the corpus arms, now limitation L8. The sound theses were self-identifying reproductions of canonical results and the flawed theses novel proposals, so a gate that passed familiar reproductions and scrutinized novel proposals would score high specificity without reading content. Pooled specificity (0.96/0.72), the sound-arm kill-rate contrast and the M1 default-flip rationale are therefore exploratory; a fresh-corpus validation must use matched-pair arms. Attribution accuracy (13/15, 14/15) is measured within the flawed arm and is unaffected.

Citation Fidelity Protocol

Three of the five v0.1.0 abandonment citations were found misread on audit (scope overclaim, a smoothing bandwidth read as a block length, a risk metric read as a validation protocol). The v0.2.0 fix is structural: every citation carries a typed claim card — claimed object type, scope conditions, and a verbatim anchor pinned by SHA-256 to its source — verified mechanically and by a dedicated citation-object-adversary role.

Limitations

Eight limitations are stated in full in the release (L1–L8; L8 added in Erratum v0.2.1), including: single run + small-n calibration (wide intervals reported in full), seeder and evaluator drawn from the same model family, named weak points (two flaw classes each detected at 1 of 2 under M1-ON), single-model dependence, the spent corpus behind the M1 default flip, historical citation defects, the retracted exhibit shipped provenance-only, and the genre confound between the corpus arms (L8).

Reproducibility

The calibration numbers are recomputed from the frozen verdicts with make calibration-analyze; make verify-citations checks every citation card and make calibration-prereg-check verifies that the pre-registration's freeze conditions still hold. make reproduce reruns the retracted v0.1.0 simulation and is expected to fail its integrity check, which confirms the retraction rather than reproducing a result. The calibration corpus, all 40 verdicts, sealed-label hashes, pre-registration, and full provenance trail for every abandonment ship in the release. Code is MIT; documentation is CC BY 4.0.

Versions and Archive

v0.1.0 was released on 2026-06-09, the date on this page. The Zenodo DOI 10.5281/zenodo.20645678 is the v0.2.0 deposit (2026-06-11), which predates the erratum; the concept DOI 10.5281/zenodo.20617297 covers all deposited versions. Erratum v0.2.1 (2026-06-13) is on GitHub as tag v0.2.1 and has no Zenodo deposit. The citation entry at the end of this page refers to the v0.2.0 deposit.

What Came Next (June–July 2026)

The v0.2.0 artifact became the foundation for a follow-on research program. Three developments followed the release:

The organization selected and designed a real research project (C1). Rather than studying its own gates, the org selected — through its own 12-candidate, four-gate selection process — a frontier project that turns the calibration idea outward: a cross-model benchmark of whether third-party LLM paper-review systems reject flawed research for the right reasons, designed for pre-registration (pre-registration drafted, not yet frozen). C1 is designed around a matched-pair, single-flaw corpus whose plants ≥2 independent non-author raters will certify (none contracted yet), three distinct evaluator families (the author's own model family is barred from every confirmatory cell), attribution scoring and a same-family seeding axis — the construction-validity controls that SoundnessBench (arXiv:2605.30329), the predecessor that established the bare deficit, structurally lacks. A directional cross-model pilot (n = 15) measured correct-reason attribution of 0.93 for Gemini 3.1 Pro and 0.47 for both GPT-5 and DeepSeek-R1. GPT-5 and Gemini straddled H1's 0.15 wrong-reason threshold, which put H1 at kill risk, so the per-system calibration card (H1″) was chosen as primary on 2026-06-23, before DeepSeek-R1 was measured and before any confirmatory data; H1 is retained as exploratory, and with three families measured its ≥2-of-3 test was indeterminate at n = 15. Confirmatory n was ratified at ~300 per arm per system. C1 is paused at its pre-registration checkpoint. A lock attempt on 2026-07-01 halted at the eligibility gate because the DeepSeek API terms had not yet been cached and checked by the program's license-review gate (an agent role), and the third slot was then reassigned to Mistral on roster-stability and data-governance grounds; as of the last logged entry (2026-07-05) the program is dormant pending one author action (an API key for the Mistral feasibility check), and nothing has been logged since. The selected roster (OpenAI GPT-5, Google Gemini, Mistral) is not frozen. External spend so far is about $6, all on directional pilots; $0 of the confirmatory budget has been spent, and nothing is frozen.

A public erratum (v0.2.1). The L8 genre confound, found by an independent F6 second review during C1's problem selection, led to the cross-arm specificity results being downgraded to exploratory on the public repository before any new freeze (see above) — so the public record matches what the program privately knows.

A second adversarial pre-review organization (System B). A separate 7-agent org now reviews each paper against the real, frozen rubric of its target venue before any human submits — deliberately inverting the round-1 prior toward high flaw-recall, with a non-removable human gate and an accept that is necessary but not sufficient (the external venue is terminal). Every verdict logs a prediction of the real outcome, calibrating the pipeline against actual peer review over time.

The system, the C1 program, and System B are tracked in the project writeup.

@software{saks_2026_20645678,
  author    = {Saks, Kacper},
  title     = {An Executable-Gate Multi-Agent Research Organization:
               Artifact, Case Study, and a Pre-Registered
               Gate-Calibration Study},
  month     = jun,
  year      = 2026,
  publisher = {Zenodo},
  version   = {v0.2.0},
  doi       = {10.5281/zenodo.20645678},
  url       = {https://doi.org/10.5281/zenodo.20645678}
}