Would I know if it were wrong?

An engineer is one click from approving an agent-written plan. The case is made up, but the shape is familiar. They asked for retry logic on a payment webhook handler. What came back is tidy: exponential backoff, a max of five attempts, idempotency keys on every retry, and a short note about logging failed deliveries to a dead-letter table that already exists. Clean prose. Numbered steps. Nothing looks sloppy.

Then one detail stops them. They happen to know for sure that the dead-letter table was renamed two months ago, and the old name is gone. The plan cites the old name with total confidence. Everything else still reads polished. If they hadn't known that one fact, the wrong table would have looked like local knowledge, not a miss.

This is an opinion piece about trusting agent output, and here's my claim. I think before you trust what an agent wrote, you should decide how you would notice a wrong answer. Feeling sure is not a check. A review that only asks "does this look right?" misses the cases where a wrong answer still looks polished.

Why polish fools the check

Wrong and ugly are easy to catch. Wrong and well written is harder.

I think agents are especially good at the second kind. They produce complete sentences, neat lists, and a confident tone even when a detail is invented or stale. The shape of the answer signals care. Your brain treats that signal as evidence the content was checked. It wasn't. The polish is a style of output, not a proof of correctness.

There's a related trap. When you ask "does this look right?", you're asking whether anything jumps out as broken. Clean formatting and plausible steps rarely jump out. The wrong dead-letter table in that made-up plan didn't look wrong. It looked specific. Specificity feels like knowledge. Without an independent way to catch the miss, a polished wrong answer sails through.

We've been reviewing more agent-written plans and diffs than we used to, and the near-misses that sting most are the ones that read finished. The messy drafts get extra eyes. The tidy ones get a nod. That is the opposite of where the risk sits.

Ask the question before you trust it

Before you approve, ask yourself: would I know if this were wrong, and how?

If the honest answer is "I'd feel uneasy" or "it wouldn't look right," you don't have a check yet. You have a vibe. Vibes fail on polished mistakes. You need a tell you could name out loud: a place the answer would break if it were wrong, that you can inspect without trusting the model that wrote it.

I think this question belongs before you ask for the answer, not only after. If you can't say in advance what would count as wrong, you're setting yourself up to grade on style. Decide the tell first. Then read for that tell. Everything else is secondary.

That doesn't mean you need a formal eval harness for every small task. It means you need one concrete way you'd catch a miss, written in your own head or on the page, before the polished draft arrives and starts arguing for itself.

Checks that count

Here are three kinds of checks I find useful. None of them needs a lab. All of them are things a person can do in a normal review.

A named expected failure. Before the agent runs, say what a wrong answer would look like in this case. "If it mentions the old dead-letter table, it's wrong." "If it adds a new queue, that's out of scope and wrong for this task." You're not predicting every bug. You're naming one failure mode you care about, so you know what to scan for when the output lands.

A fact you can verify without the model. Pick one claim in the output that exists in the world outside the answer: a table name, a config key, an API path, a file that should already be there. Look it up yourself. In the made-up plan above, the renamed table was that fact. One verified detail doesn't prove the whole plan, but one failed check is enough to stop a confident wrong draft from shipping.

A non-goal you can spot if it shows up. If you decided the work must not touch billing, or must not add a migration, read specifically for those. A wrong answer often arrives as extra helpful scope dressed as completeness. If a forbidden thing appears, you know the output failed, even if the rest still looks elegant.

I think these three beat a vague second pass over the whole document. A second pass still asks "does this look right?" A named check asks "did this particular wrong thing happen?" That's a different question, and it's the one polish can't answer for you.

A short made-up checklist for the webhook case, labelled as example only:

• Wrong if it references the old dead-letter table name.
• Wrong if it introduces a new queue or topic.
• Wrong if retries are unbounded.
• Verify the idempotency key field against the current webhook schema yourself.

Four lines. Enough to keep a polished miss from feeling like local knowledge.

What this is not

This is not a post about measuring models, and it's not a claim about how often people miss agent errors. I don't have numbers for that, and I'm not borrowing any. It's also not about writing better prompts so the agent never slips. Agents will still be wrong sometimes. The point is whether you have a way to notice.

It's adjacent to writing a useful review comment, and adjacent to placing a human checkpoint, but the focus here is earlier: whether you could catch a wrong answer at all, before you lean on confidence or clean prose.

Decide the tell first

Here's the reframe I'd leave you with. We often treat trust as a feeling that arrives after we read the output. The draft looks solid, so we trust it. When the draft is polished, that feeling shows up even when a key detail is wrong.

I think trust should wait on a prior decision: how would I know if this were wrong? If you can't answer that, you're not ready to ask for the work yet, or you're not ready to approve it. Decide the tell for "wrong" before you ask for the answer. Then read for that tell. Feeling sure can wait.

Written by the team building Gander, live review for markdown an agent is still writing.