Discussion about this post

User's avatar
Rafael Ibayev's avatar

Running the cheap checks first and then having a different model family review is a setup worth copying. The part I'd still keep a human close to is scope: whether the agent changed things nobody asked for. A reviewer model can confirm the code is fine and still miss that it wasn't the task. How does your team-lead model know what was in scope?

Halfcycle's avatar

Your Qodo example, tests that print FAILED and still pass, has a twin in our last build. A reviewer agent rejected a Stripe receiver because its test fixture was synthetic and skipped the escaping the spec required, so the test proved less than it claimed. It was catchable for the reason you quote from Anush: what passing meant was written down before any code, so the reviewer had something to check besides the diff. The one place I would push your loop earlier is the plan. On that build, a fresh reviewer on the spec found far more than review of the code did. We built Halfcycle around that step (npx halfcycle, inside Claude Code). Do your cross-family reviewers ever see the requirements before the code exists, or only alongside the diff?

7 more comments...

Ready for more?