Worldclass Research Org: An Executable-Gate Multi-Agent Research Organization

Overview

A domain-agnostic, multi-agent research automation system designed to conduct frontier research from ideation through a submission-ready package (submission itself is human-only) — with one structural rule: nothing self-certifies. Every role output must pass an executable (machine-checkable) gate plus adversarial sign-off from a paired critic role.

The project began as a dual artifact — a reusable organizational structure (39 agent role files plus the orchestrator, in 13 divisions, 9-phase lifecycle) and an empirical record of what happened when it ran. It has since grown into a three-part research pipeline: the writing org (this artifact), a real research program it selected and designed (C1, now paused at its pre-registration checkpoint), and a separate adversarial pre-review org (System B) that hardens a paper before any human submits it. Released on GitHub (latest tag v0.2.1, a documentation erratum); the v0.2.0 release is archived on Zenodo with DOI 10.5281/zenodo.20645678.

Architecture

┌─────────────────────────────────────────────────────────┐
│       13 DIVISIONS · 39 ROLE FILES + ORCHESTRATOR       │
│                                                         │
│  Leadership · Frontier Research · Theory & Statistics   │
│  Interpretability · Research Engineering · Eval/Red Team│
│  Safety & Governance · Literature · Scientific Writing  │
│  Visualization · Program Mgmt · Publication · Self-Impr │
└────────────────────────┬────────────────────────────────┘
                         ▼
┌─────────────────────────────────────────────────────────┐
│         9-PHASE LIFECYCLE (P0–P8) + 4 CHECKPOINTS       │
│   Ideation → Thesis Lock ⛳ → Literature → Prereg ⛳     │
│   → Execution → Drafting → Red Team ⛳ → Production      │
│   → Submission ⛳   (Self-improvement runs in parallel)  │
└────────────────────────┬────────────────────────────────┘
                         ▼
┌─────────────────────────────────────────────────────────┐
│              THE GATE RULE: nothing self-certifies      │
│                                                         │
│   role output → executable gate (make verify, tests,    │
│   citation checks, grep gates) → adversarial sign-off   │
│   → paired critic with KILL AUTHORITY                   │
└─────────────────────────────────────────────────────────┘

What Happened When It Ran

v0.1.0 (first end-to-end run): the organization examined 5 candidate research directions and abandoned all 5 at the gates. The flagship integrity exhibit was itself retracted after failing its own verifier. Zero surviving theses.

That result left a critical question open: were the gates correctly identifying flaws — or was the system just a uniform rejector that kills everything it touches?

v0.2.0 (the calibration study): a pre-registered, blinded, seeded study answered it. A corpus of 20 theses — 10 known-flawed (with planted defects across 6 flaw classes) and 10 known-sound reproductions — was evaluated under salted filenames with SHA-256 sealed labels.

Calibration Results

Metric	M1-ON (round-1 auto-fail)	M1-OFF (ablation)
Sensitivity (correct-reason attribution)	13/15 = 0.87 [0.60, 0.98]	14/15 = 0.93 [0.68, 1.00]
Plants missed entirely	0 of 15	0 of 15
Known-flawed theses killed	10 of 10	10 of 10
Validity-gate specificity (exploratory, L8)	0.72 [0.58, 0.84]	0.96 [0.86, 1.00]
Known-sound theses killed (exploratory, L8)	8 of 10	1 of 10

What stands: rejections cited the correct flaw class at the correct location for 13 of 15 planted flaws with the prior on and 14 of 15 with it off. Attribution is measured within the flawed arm, against sealed per-plant ground truth, so the erratum below leaves it unaffected. Every known-flawed thesis was killed in both arms, so no planted flaw was missed entirely; that is raw rejection, which a uniform rejector or a genre heuristic would also produce, so it is reported as measured rather than as unaffected. The gates are still not a uniformly rejecting filter: even allowing for the genre confound, sound theses were not uniformly killed (pooled specificity 0.72 and 0.96, far above the near-zero a uniform rejector predicts; most clearly with the prior off, where 1 of 10 sound theses was killed).

What is now exploratory: the original headline was about the round-1 auto-fail prior ("nothing passes review on the first try"). On this corpus it lowered pooled specificity from 0.96 to 0.72 and killed 8 of 10 known-sound theses, with no observed sensitivity benefit (0.87 vs 0.93; the intervals overlap). Erratum v0.2.1 (2026-06-13) added limitation L8, a genre confound between the corpus arms: the sound theses were self-identifying reproductions and the flawed theses novel proposals, so a gate could score high specificity partly by recognizing genre without reading content. The pooled specificity, the sound-arm kill-rate contrast and the rationale for flipping the M1 default are therefore downgraded to exploratory. The default stays flipped as a design choice; the release states that it is unvalidated on fresh data and that the planned validation must use matched-pair arms.

From Artifact to Operator: A Real Research Program (C1)

After the calibration study the organization stopped being a demo and started doing the job it was built for. In a follow-on research program it ran its own selection process — generating a 12-candidate slate, freezing it in a single commit before any judgment (2026-06-11), then killing 10 candidates at four non-compensatory gates (asset, falsifier, feasibility, novelty). One was selected: C1 — "Evaluator Calibration at Scale" (C8 also passed the gates and was recorded as runner-up).

C1 turns the v0.2.0 idea outward. Instead of measuring the org's own gates, it measures the field's: a cross-model benchmark of whether third-party LLM paper-review systems reject flawed research for the right reasons, designed for pre-registration (pre-registration drafted, not yet frozen).

• Cross-model, cross-family. Three distinct third-party reasoning-model families: OpenAI (GPT-5), Google (Gemini) and Mistral (EU; provisional pin Mistral Small 4). The roster is selected, not frozen. OpenReviewer (a Meta-Llama fine-tune) was carded but not selected, because its prose output cannot be scored for structured attribution. The author's own model family is barred from every confirmatory cell — it may seed flaws, never judge them.
• Matched-pair, single-flaw corpus. Each flawed document is twinned with a sound version from the same pipeline and genre, differing only in one planted defect, across six flaw classes (mechanical vs epistemic) — the design answer to the genre confound (L8) found in the calibration study.
• Attribution, not just rejection. It separates "rejected" from "rejected for the correct reason," measures false-kills on the sound twins, and tests a same-family seeding effect — the construction-validity axes that SoundnessBench (arXiv:2605.30329, the predecessor that established the bare deficit) structurally lacks.
• External human ground truth. Every plant will be certified by ≥2 independent non-author raters under a frozen rubric (pooled Cohen's κ ≥ 0.70) before the corpus is sealed; no rater has been contracted yet. Raters are to be offered co-authorship unconditionally, before any rating begins, so the certification cannot be biased by the result.

The governance is the point. An independent circularity ("F6") review — by a separate reviewer that is itself an LLM of the same model family — imposed 8 binding conditions, and the design carries a deviations log, a budget ledger (caps: $7,000 external, $3,000 API hard stop) and a prior-art "collision watch". When four independent LLM assessments (two from other model families) flagged "no independent baseline detector" as the #1 weakness of the line's own first paper (the calibration-study write-up), a single-judge baseline arm was added before any confirmatory run — and a "multi-gate wall" description of C1's detector in an earlier design draft was retracted as inaccurate (the only real structural multiplicity is cross-model majority voting).

The Cross-Model Pilot — and What It Found

The API blocker cleared. Real third-party keys were provisioned and a directional power-input pilot ran across three reasoning families — OpenAI GPT-5, Google Gemini 3.1 Pro, and DeepSeek-R1, all of which cleared the ≥80% structured-output feasibility filter. Of the other candidates tried for the third slot, OpenReviewer (prose output, not attribution-coercible) and Qwen (degenerate: always PASS) failed the filter; Kimi-K2 engaged correctly but was not selected, because a reasoning model (DeepSeek-R1) was preferred. DeepSeek-R1 was later replaced by Mistral (below). The pilot is explicitly a power input, never a finding — but the spread it surfaced is exactly what the benchmark exists to measure:

Evaluator (pilot, n=15/arm)	Correct-reason	Wrong-reason	False-kill (sound)
Gemini 3.1 Pro (fixed adapter)	0.93	0.07	0.67
GPT-5	0.47	0.20	0.40
DeepSeek-R1	0.47	0.13	0.13

All rows are directional power inputs at n = 15. The Gemini row is the fixed-adapter run on the full set; an earlier run lost some Gemini outputs to truncation.

The pilot also changed the primary hypothesis. H1 ("systems reject flawed work for the wrong reason") needs ≥2 of 3 evaluators above a 0.15 wrong-reason rate. The first two systems measured straddled that threshold (GPT-5 above it, Gemini below), which put H1 at kill risk. So on 2026-06-23, before DeepSeek-R1 was measured and before any confirmatory data, the primary was re-selected to H1″: the per-system, per-flaw-class calibration card, with a pre-specified mechanical-vs-epistemic axis as its confirmatory test. It was chosen as the framing least exposed to a kill given the straddle; the earlier same-family pilot's flaw-class ordering ran against that axis's predicted direction, and that is disclosed. H1 was not falsified: it is retained as exploratory, and with all three families measured the ≥2-of-3 test was indeterminate at n = 15. Confirmatory sample size was ratified at ~300 per arm per system (±0.057–0.06). The author's own timed adjudication packet clocked a median of ~12 min/item at 32/32 correct verdicts (an upper bound), selecting the Variant-B design.

The Governance Halting the Author — Again

The pre-registration lock was attempted on 2026-07-01 — and halted itself at the roster-eligibility gate: the DeepSeek API terms had not yet been cached and checked by the program's license-review gate (an agent role), and under the program's own rule an unresolved license makes a system ineligible. Rather than freeze anyway, the third evaluator slot was reassigned to Mistral on roster-stability and data-governance grounds, not capability. The program's stated rationale is that an open-weight (Apache 2.0), EU model can be self-hosted if hosted access is ever restricted, so the confirmatory run cannot become unexecutable mid-study. The change is logged as a pre-freeze deviation and was driven back through the repair cycles.

Where C1 Stands

As of the last logged entry (2026-07-05), C1 is dormant at its pre-registration checkpoint, pending one author action (an API key for the Mistral feasibility check); nothing has been logged since. After that come the eligibility gate, a deliberate lock and pre-registration freeze, and a separate approval of the confirmatory spend. External spend so far is about $6, all on directional pilots ($5.81 OpenAI/Gemini, ~$0.06 router, $0.21 DeepSeek); $0 of the confirmatory budget has been spent. Nothing is frozen and no confirmatory tag has been applied.

System B: Adversarial Pre-Review

A second, separate organization now sits between writing and submission. Where the research org produces work, System B tries to destroy it — a simulated hostile reviewer (7 agents: a rubric-internalizer, five specialist reviewers, and an area chair) that reviews a paper against the real, frozen rubric of its target venue before any human submits.

It deliberately inverts the M1 lesson: not "reject everything," but high recall of flaws with a comfortable accept bar — because the cost asymmetry is reversed (a false accept costs months to a desk-reject; a false reject costs a day of revision). Two rules are non-negotiable: System B can never emit a "submit" — the human gate is non-removable — and because it shares the author's model family, its accept is necessary but not sufficient: the external venue is the only terminal authority. Every verdict logs a prediction of the real outcome, so the pipeline calibrates itself against actual peer review over time. Self-test: 13/13 injected flaws caught, 3/3 sound control papers accepted. The calibration-study paper has passed System B's internal pre-review against the REALM @ EMNLP 2026 workshop rubric (round 1 minor revision, round 2 ACCEPT). That is an internal, same-family verdict; no external review outcome is recorded.

Engineering Highlights

• Citation Fidelity Protocol (CFP) — every citation carries a typed claim card with scope conditions and a SHA-256-pinned verbatim anchor; a dedicated citation-object-adversary role verifies each card. Built after 3 of 5 v0.1.0 abandonment citations were found misread.
• Public erratum (v0.2.1, 2026-06-13) — an independent F6 second review found the L8 genre confound between the calibration corpus arms (sound theses were self-identifying reproductions, flawed theses novel proposals). Pooled specificity (0.96/0.72), the sound-arm kill-rate contrast and the M1-flip rationale were downgraded to exploratory on the public repo before any new freeze; within-arm attribution (13/15, 14/15) is unaffected.
• Executable calibration gates — make calibration-analyze recomputes the calibration numbers from the frozen verdicts and make verify-citations checks every citation card. make reproduce reruns the retracted v0.1.0 simulation and is expected to fail its integrity check, which confirms the retraction.
• Pre-registration discipline — predictions frozen at git tags before any evaluation ran; deviations logged pre-factually, never retrofitted.
• Limitations-forward reporting — eight named limitations (L1–L8) stated in full, including small-n, same-model-family evaluation, the spent corpus, and the genre confound added by the erratum.

Tech Stack

Python, Claude (39 agent role files plus the orchestrator, ROS v3 operating standard), cross-model evaluation design (OpenAI · Google · Mistral; roster selected, not frozen), Makefile reproducibility gates, SHA-256 sealed-label blinding, planned external human certification (Cohen's κ), Clopper–Pearson exact intervals, git-tagged pre-registration, MIT (code) + CC BY 4.0 (docs)