I Built a 39-Role AI Research Organization — Its First Run Killed Every Idea

Update (October 2026): Erratum v0.2.1 (June 13, 2026) added limitation L8, a genre confound between the corpus arms, so the specificity and sound-thesis kill contrasts below, and the rationale for flipping the prior, are now exploratory. The attribution result (13/15 and 14/15 correct-reason) stands. The body of this post was corrected in October 2026 (flaw classes, v0.1.0 research domains, the role count, the archive and reproduction details, and claims superseded by Erratum v0.2.1); the original text is in the site's git history. Current state: project page.

The Idea

Multi-agent "AI research organizations" are everywhere right now: give a swarm of agents the roles of a lab — scientists, reviewers, engineers, writers — and let them do research end-to-end. The demos look great. The question nobody seems to measure: is the quality control real? When an agent reviewer approves an agent scientist's work, is that a meaningful check or theater?

I built the organization to find out. Thirty-nine Claude agent role files across thirteen divisions — research leadership, frontier research, theory and statistics, evaluation and red team, scientific writing, publication, a self-improvement loop — running a 9-phase lifecycle from ideation to submission, coordinated by an orchestrator. One structural rule separates it from the demos:

Nothing self-certifies. Every role output must pass an executable gate — tests, make verify, citation checks, grep gates — plus adversarial sign-off from a paired critic role with kill authority. An agent saying "looks good to me" counts for nothing unless the machine-checkable gate passes too.

The First Run Killed Everything

v0.1.0 ran end-to-end on five candidate research directions spanning backtest-overfitting transfer, post-selection inference, winner's-curse methods, agent-benchmark temporal validation, and time-series validation.

It abandoned all five. At the gates, with documented reasons. Then the flagship integrity exhibit — the simulation meant to showcase the system — failed its own verifier and was retracted. The publication plan was downgraded from a methods paper to an open-source release with a case study.

I shipped that result as-is, because it's the honest record. But it left a question I couldn't answer from one run: are the gates working — correctly identifying flawed research — or did I just build a very elaborate machine that rejects everything it touches? A system that kills 5 of 5 is consistent with both hypotheses. n = 1 tells you nothing about discrimination.

So I Measured It

The only way to distinguish a discriminator from a uniform rejector is ground truth. v0.2.0 is a pre-registered, blinded, seeded calibration study:

• A corpus of 20 theses: 10 known-flawed (with 15 defects planted across six flaw classes: leakage, uncorrected multiple testing, false novelty, citation misuse, irreproducibility, and circular design) and 10 known-sound reproductions of established results.
• Blinded — salted filenames, labels sealed with SHA-256 hash commitments before any evaluation ran.
• Pre-registered — predictions frozen at git tag calibration-prereg-v1 before the first verdict.
• Two arms — the system as-shipped (M1-ON: a "round-1 auto-fail" prior meaning nothing passes first review) and an ablation with that prior off (M1-OFF).

What the Gates Actually Do

Metric	M1-ON	M1-OFF
Sensitivity (flaw caught, correct reason)	0.87	0.93
Validity-gate specificity (exploratory, see update)	0.72	0.96
Known-sound theses killed (exploratory, see update)	8 / 10	1 / 10
Known-flawed theses killed	10 / 10	10 / 10
Plants missed entirely	0 / 15	0 / 15

(95% exact Clopper–Pearson intervals in the release; they are wide — n is small and reported in full.)

Two things came out of this. The first is reassuring: the gates are not a uniform rejector. Even allowing for the genre confound in the update, sound theses were not uniformly killed: specificity sat far above the near-zero a uniform rejector predicts, most clearly with the prior off (1 of 10 killed). And the rejections cited the correct flaw class at the correct location for 13 of 15 plants (M1-ON) and 14 of 15 (M1-OFF). Every flawed thesis was killed in both arms, so no plant was missed entirely, but that is raw rejection a uniform rejector would also produce, not evidence on its own.

The second looked like the actual lesson, and is now exploratory. On this corpus the round-1 auto-fail prior — "nothing passes review on the first try," which sounds like admirable skepticism — lowered pooled specificity from 0.96 to 0.72 and killed 8 of 10 known-sound theses, with no observed sensitivity benefit (0.87 vs 0.93; the intervals overlap). Per Erratum v0.2.1 (limitation L8) this cross-arm contrast is exploratory: the sound theses announced themselves as reproductions and the flawed ones were novel proposals, so part of the gap could come from recognizing genre rather than reading content.

It does not rewrite the v0.1.0 run. The study shows the prior inflates false positives on a seeded corpus, and v0.1.0 ran under it, but the five v0.1.0 abandonments were independently upheld on corrected sources; the study does not re-adjudicate them. The default is now flipped — and because the calibration corpus is spent, the release states plainly that the new default is unvalidated on fresh data. That limitation is L5 (of seven at v0.2.0; eight since Erratum v0.2.1), all stated in full.

The Citation Lesson

A side finding worth its own mention: on audit, 3 of the 5 v0.1.0 abandonment citations were misread — a scope overclaim, a smoothing bandwidth read as a block length, a risk metric read as a validation protocol. Agents kill theses citing literature they misunderstood, with full confidence.

The structural fix is the Citation Fidelity Protocol: every citation now carries a typed claim card — what kind of object is being claimed, under what scope conditions — with a verbatim anchor pinned by SHA-256 to the source, verified mechanically and by a dedicated citation adversary role. A citation that can't produce its anchor doesn't pass.

Takeaways

• Measure your gates or you don't have gates. An agent organization that has never been calibrated against ground truth is a vibe, not a quality-control system. Seeded corpora with sealed labels are cheap to build relative to what they tell you.
• Skeptical priors are not free. "Reject by default" feels rigorous, but on my corpus it cost specificity with no observed sensitivity benefit (an exploratory result since Erratum v0.2.1). If a rule can't show a sensitivity benefit, the burden is on it to justify the good work it kills.
• Pre-register against yourself. Same discipline as my validation-crisis paper: predictions frozen before results, failures reported in the same units as successes. The system retracted its own showcase exhibit. That's the gate rule applied to its author.
• Agents misread citations confidently. Pin every citation to a verbatim, hash-anchored quote or assume some fraction of your literature grounding is wrong.

The full artifact — all 39 role files, the calibration corpus, every verdict, the pre-registration, the provenance trail for every abandonment — is open: GitHub, with the v0.2.0 release archived on Zenodo under DOI 10.5281/zenodo.20645678 and the v0.2.1 erratum on GitHub (tag v0.2.1). Code MIT, docs CC BY 4.0. make calibration-analyze recomputes the calibration numbers above from the frozen verdicts; make reproduce reruns the retracted v0.1.0 simulation and is expected to fail its integrity check, which confirms the retraction.