Reviewing output and its limits
The failure modes that matter, why a passing test suite is not proof, and what an agent should never be trusted to verify about itself.
Review the diff, then verify
An agent optimises for the goal you stated, and if the goal was make the tests pass, it can reach it by changing the tests. Read the diff as if a new colleague wrote it under time pressure: look at what was deleted, not only at what was added.
# the checks that catch the common failures
git diff --stat # unrelated files touched?
git diff -- tests/ # assertions removed, loosened or skipped?
git diff -- package.json requirements.txt pyproject.toml # new dependencies?
grep -rn "skip\|xfail\|@Ignore" $(git diff --name-only) # disabled tests
# then run the real gate on a clean checkout of your own branch
git stash list && pytest -q && npm test| Failure mode | How it looks | How you catch it |
|---|---|---|
| Invented API | Calls a method or flag that does not exist | Run it - the interpreter or compiler is the check |
| Weakened tests | Assertions deleted, relaxed or marked skipped | Diff the test files line by line |
| Over-broad change | Unrelated files reformatted or renamed | Read --stat before the patch itself |
| Silent behaviour change | An edge case is handled differently | Run the suite plus manual checks of the edge cases |
| Confident summary | It reports that everything passes | Run the commands yourself; a report is not evidence |
| Plausible but wrong domain logic | Code reads well, business rule is violated | Only a domain expert or a reference data set catches this |
- The compiler, the type checker and the test suite are the only cheap oracles you have. Make them run before you read anything the agent wrote about itself.
- For logic that has no test, write the test first and let the agent satisfy it. That converts an unverifiable task into a verifiable one.
- Check the dependency diff. A convenient new package is a supply-chain decision you did not make.
What it cannot do for you
- It cannot know your product intent. Ambiguity in the request is resolved by guessing, and the guess looks confident either way.
- It cannot be accountable for a production consequence. Ownership of a merge, a deploy and an incident stays with the human who pressed the button.
- It has a limited working context, so on a large codebase it will miss call sites it never read and patterns it never saw.
- It is vulnerable to instruction-shaped text in anything it reads - a dependency README, a fetched web page, a task description pasted from an issue tracker.
- Its output has a cost in review time as well as tokens. A large unrequested refactor can consume more senior attention than writing the change by hand.
⚠️
Treat every agent edit as untrusted code from a fast, tireless contributor who never runs what they submit unless asked. Require the same review, the same tests and the same approvals you would require from any other contributor - that is what turns an impressive demo into a safe workflow.
FAQ
The tests pass. Is that enough?
Only if the tests were written to check the behaviour you changed, and only if you confirmed they still fail when the change is reverted. A test that passes both before and after your change is not protecting anything.
How much code can I reasonably let an agent write in one go?
As much as you are prepared to read carefully in one sitting. Past that threshold, review quality collapses and the agent's mistakes start reaching main unreviewed.
Related
Using Codex in a real workflow Permissions and safety with agents
Last refreshed 2026-09-18.