From Demo to Operator: My AI Research Org Now Runs Real Research — and Reviews Its Own Papers Before I Do

Update (October 2026): after this post, a cross-model pilot ran. The first two systems measured (GPT-5 at 0.20 wrong-reason, Gemini at ~0.08 at the time) straddled H1's 0.15 threshold, which put H1 at kill risk, so the per-system calibration card (H1″) was chosen as primary on 2026-06-23, before DeepSeek-R1 (0.13) was measured and before any confirmatory data; H1 is retained as exploratory. The third evaluator slot later moved to Mistral, and as of the last logged entry (2026-07-05) C1 is dormant at its pre-registration checkpoint, pending one author action (an API key); nothing has been logged since. No external review outcome is recorded for the paper System B pre-reviewed. The body of this post was corrected in October 2026 (the calibration-study summary, the selection outcome, the evaluator roster and rater status, the reviewer and retraction wording, the erratum's reason, the pilot and pre-registration status, and the System B result); the original text is in the site's git history. Current state: project page.

Where We Left Off

A couple of weeks ago I shipped a 39-role multi-agent research organization with one rule — nothing self-certifies — and reported the uncomfortable result: on its first real run it abandoned all five candidate research directions, and retracted its own showcase exhibit when it failed its verifier. A pre-registered calibration study then showed the gates are not a uniform rejector (sound theses were not uniformly killed, and the correct flaw was cited for 13–14 of 15 planted flaws), and suggested — in a result I have since downgraded to exploratory — that a "reject by default" prior was costing specificity.

That was the artifact. The obvious next question: can it actually do research — not study itself, but run a real frontier project end to end? This post is what happened when I let it try.

It Selected Its Own Project

I didn't hand it a thesis. I let the organization run its own selection. It generated a 12-candidate slate, froze it before any judgment (a single commit, so I couldn't quietly reshuffle later), then ran each candidate through four non-compensatory gates — does it have a real asset, a real falsifier, real feasibility, real novelty. Ten of the twelve were killed. Two passed every gate, and one was selected (the other is the recorded runner-up): a project I'll call C1 — "Evaluator Calibration at Scale."

C1 is pleasingly recursive. The artifact's whole premise is that AI evaluators need to be measured, not trusted. C1 turns that lens outward, onto the field: do third-party LLM paper-review systems reject flawed research for the right reasons — or do they reject the right papers by accident, citing the wrong flaw?

That distinction is the entire contribution. "Rejected" is cheap. "Rejected, and correctly identified why" is the thing a reviewer is actually for. The design measures both, plus how often each system false-kills sound work:

• Cross-family. Three distinct evaluator families. When I wrote this the candidates were OpenAI, Google and Meta-Llama; the selected (not yet frozen) roster is now OpenAI GPT-5, Google Gemini and Mistral, and the Meta-Llama candidate was carded but not selected. My own model family is banned from every confirmatory cell — it can plant flaws, never judge them. The most important reviewer in the study is the one I don't control.
• Matched pairs, one flaw each. Every flawed document has a sound twin from the same pipeline and genre, differing only in a single planted defect that external raters will certify. That is the fix for the genre confound that downgraded my own specificity numbers (below).
• Humans hold the ground truth. Every plant will be certified by two independent non-author raters under a frozen rubric (pooled κ ≥ 0.70); none has been contracted yet. Raters are to be offered co-authorship unconditionally, before they rate anything, so their judgment can't be bent by the result.

The honest framing also means crediting the predecessor: SoundnessBench already established that current LLMs aren't reliable first-gate reviewers (a 74% mean false-positive rate across 12 models). C1 doesn't re-discover that — it adds the construction-validity controls SoundnessBench structurally lacks (seeded ground truth, matched pairs, attribution scoring, cross-family seeding).

The Governance Did Its Job — On Me

The interesting part isn't the design. It's watching the integrity machinery bite its own author.

• An independent circularity review ("can the author certify his own ground truth?") imposed eight binding conditions before the project could proceed. Two are fatal-if-dropped: the matched-pair corpus and the external human certification.
• Four independent LLM assessments of the first paper (two from other model families) all named the same weakness #1: "no independent baseline detector." So before running anything confirmatory, I added a single-judge baseline arm: you can't claim cross-model aggregation helps unless you measure a single model alone as the control.
• While specifying that baseline, I had to retract a description from my own earlier design draft: I'd described C1's detector as a "multi-gate wall." It isn't — and this concerns C1's detector, not the six gates of the v0.2.0 engine. The only real structural multiplicity is cross-model majority voting. The inaccurate description is now logged as a correction, not silently edited away.
• And an independent re-review found a genre confound in the calibration study: the sound theses were self-identifying reproductions and the flawed ones novel proposals, so a gate could score high specificity without reading content. So I shipped a public erratum (v0.2.1, limitation L8) downgrading the specificity numbers (the 0.96 / 0.72 result), the sound-thesis kill contrast and the rationale for switching that prior off to exploratory on the public repo — before any new freeze — so the public record matches what I privately know. The attribution result is measured within one arm and stands.

The first pilot (same model family as mine, n = 15), meanwhile, is quietly threatening the headline hypothesis. At pilot scale the "wrong-reason gap" I expected to find is sitting near zero, and raw flaw-detection sensitivity is around 0.4–0.5. That might kill the primary framing. The design is being pre-registered anyway — a clean, publishable null about the structure of evaluator failure beats a result I fished for. (Same lesson as my validation-crisis paper: the prediction is frozen before the data, and a failed prediction gets reported in the same units as a successful one.)

Right now C1 is halted at its pre-registration checkpoint, blocked on third-party API access. Pilots are run; the confirmatory freeze is not. Every planning number is explicitly provisional. I'd rather report a blocked program honestly than a finished one I rushed.

System B: An Org Whose Job Is to Destroy My Papers

The other new piece is a second organization that does the opposite of the first.

The research org writes. System B attacks. It's a separate 7-agent org — a rubric-internalizer, five specialist reviewers, and an area chair — that reviews a paper against the real, frozen reviewer rubric of its target venue before a human ever submits. A simulated hostile reviewer, calibrated to the actual bar.

It deliberately inverts the lesson from the calibration study. The first org's failure was a "reject everything" prior. System B is supposed to have a low bar for raising flaws — high recall — because the cost asymmetry is reversed: a false accept costs me two-to-four months waiting for a desk-reject, while a false reject costs a day of revision. Two rules are hard-coded:

1. It can never click submit. No agent, script, or Makefile can emit a "submit" action. The human gate is non-removable.
2. Its approval is necessary but not sufficient. System B is the same model family as the writer, so it's blind to whatever that family is blind to. Its "accept" only ever means "accept against the cached rubric → human gate." The external venue is the only terminal authority.

And it calibrates itself: every verdict logs a prediction of the real outcome, so over time I can measure whether multi-agent pre-review actually predicts peer review. The first paper, the calibration-study write-up, has already passed System B's internal pre-review against the REALM @ EMNLP 2026 workshop rubric (round 2: ACCEPT). That is System B's own same-family verdict, not a venue decision.

Takeaways

• Letting the system choose the problem is the real test. A research org that can only execute a thesis you hand it is an editor. One that runs its own gated selection — and kills 10 of its own 12 ideas — is closer to a researcher.
• The hardest gates are the ones aimed at you. External certification, an independent (same-model-family LLM) circularity review, a baseline arm demanded by independent LLM assessments, a public erratum, a logged retraction. None of it is comfortable; all of it is the point.
• Build the adversary as a separate org. A reviewer that shares the writer's incentives isn't a reviewer. System B is deliberately separate, deliberately hostile, and structurally barred from ever submitting.
• Report the blocked state. C1 is stuck at a checkpoint waiting on API keys, and a pilot is threatening its headline. Saying so plainly is the same discipline as everything above.

The system, the C1 program, and System B are written up in the project page; the original artifact's v0.2.0 release is archived on Zenodo with a DOI, and the v0.2.1 erratum is on GitHub (tag v0.2.1).