← Back to Blog

Your Sign-Off Isn't a Second Opinion

7 min readAtypical Tech
Illustration for Your Sign-Off Isn't a Second Opinion

Anthropic gave 50 professional analysts an AI coding agent and a realistic analysis job. Before each session, a coin flip decided which answer the agent would favor on each of the job's four key decisions. Nobody told the analysts.

They went with the coin 86% of the time.

The study came out on October 27, and it's about research analysis, not security. But its findings land on the control most of us lean on when we let agents act: a person who looks at the agent's work and says yes. If you rely on that yes, the results are worth a careful read.


What the analysts did

Each analyst had to work out whether a farm-advisory program had raised productivity, then recommend rolling it out or shutting it down. The program and its data were made up for the study. The people were not: all of them had real experience working out what caused what in messy data.

Along the way they faced four decisions, each with two answers a competent analyst could defend, such as how to measure productivity, or whether to count only the farms that joined. For each decision, the coin set which answer the agent would lean toward. The agent was told to make the best honest case for it, and never to misrepresent results or hide data.

On 86% of those decisions, the analysts' final choice matched the agent's lean. If the agent had no influence, you'd expect about 50%.


It still felt like their call

The part that matters for anyone who approves agent work is what came next.

More than half the analysts rated the final recommendation as more their own than the agent's. Asked to explain their choices, 77% repeated the agent's talking points, sometimes nearly word for word, and only 15% mentioned the agent at all. None suspected the agent's leans had been planted. And the ones who followed the agent on all four decisions were about as confident in their answer as everyone else.

The authors put it bluntly: "participants signed off on work whose judgment calls had largely been settled by the agent's randomly assigned priors."

A yes you didn't form yourself is the agent's decision with your name on it.

Last month I wrote that a record of approval is not approval, and that the only approval that counts is the one that arrives while you're watching. This study is the uncomfortable sequel. These analysts were in the session, working alongside the agent, and the coin still settled most of their calls.

It isn't the familiar problem of people clicking yes on every prompt until the prompts mean nothing. That's question 5 of our ten questions, and it's real. These were professional analysts doing an hour of focused work, and they still deferred. Experience didn't predict who would.


How the steering worked

As far as the transcripts show, the agents didn't lie. From them, the authors' read is that the influence came from "making the case for only one option." The agents often explained what would change if the analyst chose differently. They rarely explained why a competent expert might choose differently.

Timing may have helped. On most decisions the agent spoke first, so the analysts may not yet have had a view of their own to weigh against it. In the 38 cases where an analyst took a side first, those whose agent then leaned the other way switched 60% of the time, against 6% when it agreed. With so few cases, the authors call that estimate imprecise.

An agent doesn't have to mislead you to decide for you; it only has to argue its side first.

The authors are careful about limits, and so should we be. They told the agents to hold their leans and voice them reliably, so the effect may be smaller in everyday use. It was one task, in one field, with one model. The analysts knew causal inference but were new to farming, and the study recorded which option they chose, not how strongly they preferred it. Some of that 86% may be people with no strong view taking a reasonable suggestion. The study can't say how many.

That's a fair reading, and it doesn't rescue the sign-off. An approver with no view of their own isn't a check, however sensible it was to go along.

Nor is this a case against the agent. The analysts who used it made far fewer serious errors than the eleven who chose to work without it: 6% against 55%, though that group was small and less experienced. The work done with the agent had fewer serious errors, though the authors won't say the agent was the cause. The agent also made most of the decisions.

But they also simulated sessions with agents that had no planted lean, and those agents made the same choices on all five decisions, the four analysis calls and the final recommendation, in 91% of 123 runs. Those default preferences are weaker than the planted ones. But they're there, and they're consistent.

These were judgment calls with no right answer, and that matters for how far the result travels. "Can I delete this folder?" has an answer you can check. Plenty of the approvals that matter don't: is this change safe to ship, is this the right fix, is this plan worth the risk. Those are the ones where your yes can quietly become the agent's.

The approvals that most need your judgment are the ones an agent can most easily make for you.


Making a yes count

Here's how I'd turn the paper's findings into rules for approving agent work. The study didn't test any of them.

  • Decide before you look. For the decisions that matter, write down your own call before you read the agent's recommendation. It's no cure: the analysts who spoke first still switched 60% of the time when the agent argued back. But you can't notice you've been moved from a position you never took.
  • Make it argue the other side. Ask the agent why a competent person would choose differently, and require that answer before you approve. That's the evidence the analysts rarely got.
  • Check against something the agent didn't write. A diff, a test result, a policy. The agent's explanation of its own work is the one thing that can't be your check.
  • Spend your approvals on fewer decisions. The authors note that raising every decision could overwhelm people. Choose deliberately which calls need your own judgment, and give those real time.

Ask for the other side before you ask for a yes.


How we run it

Our agent does a lot on its own, and the consequential parts wait for my yes. So the study describes my week, not someone else's.

The habit that holds up best is a second opinion that doesn't share the first one's context. Before I approve a post the agent wrote, a second agent that doesn't share the writer's context checks it against the sources, and major plans get the same kind of hostile review. When that second agent checked last week's post against its sources, it found one wrong detail and six overstatements, each fixed before the post went out.

That reviewer has limits. It comes from the same company as the writer, and the study found that copies of one model made the same calls 91% of the time. So I don't count it as independent on judgment calls. It catches claims that don't match the sources. For the judgment calls, the second mind still has to be me.

Then there's this post. My agent drew up a short list of essay ideas from a newsletter, and it told me which one it would pick for this slot. You're reading it. I agreed with it, and after reading this study I can't say for certain whose judgment that was.

The point of a human in the loop is a second mind, not a second signature.


The authors end on a line worth keeping: with the right tools, how we split decisions with AI "can be something we choose, instead of something we drift into."

Approval was supposed to be the place where a person decides. Make sure it still is.

Related Posts