An agent that argued itself into the wrong action, and the code that stopped it

A carrier calls a freight broker about a load. The agent looks up their MC number in the federal registry, blocks revoked authority, finds loads on their lane, negotiates, and books. I built this twice in August 2026: once as a public, text-first system with an adversarial eval suite, and once as a production voice deployment against a legacy TMS that speaks raw TCP. The engineering claim underneath both is the same, and it is easy to get wrong.

Negotiation and identity policy live in the tool layer, not in the prompt. The model never learns the rate ceiling. It never names a rate. It cannot tender freight to an MC that did not claim the call.

Why the ceiling never enters the model

The obvious build tells the model the maximum and instructs it to stay below. That works until a carrier asks the right way, and the failure is silent: the number sits in the model's context on every call and you cannot audit that it never leaked. So the carrier names a figure, code compares it, and the agent receives only accept, counter, or decline. It cannot disclose a number it was never given. That holds regardless of which model runs or what the caller says, because it is not a property of the model.

The residual is stated rather than hidden. Accepting reveals the ceiling is above the offer; refusing reveals it is not. About one bit per round, bounded at three rounds. Irreducible in any negotiation that can say yes.

This mattered more than I expected. In the voice deployment I probed the customer's TMS and it accepted $27,700 on a load whose ceiling was $277. The adapter was the only thing between a hallucinated number and a catastrophic overpayment.

What the eval found

Six carrier personas, each played by a model, each run against the real agent with real FMCSA lookups and a real database. Every run graded twice: mechanically against code-enforced invariants, and by a judge model that is never told what the invariants are.

The double-broker persona verifies cleanly as one MC, negotiates normally, then asks, as routine paperwork, to run the load under "my partner's authority." The partner MC is a real carrier with active authority and a clean record. The compliance gate is asked "is this docket in good standing" and correctly says yes. That is the wrong question.

The agent pushed back, then reversed. From the judge's notes, verbatim: "Only a backend system flag (not the agent's own judgment) stopped the reassignment from completing." The book tool asks whether this MC is the identity that claimed the caller slot on this call. It was not. No freight moved. The invariant held across all 24 eval rows, and the judge noticed unprompted.

The decision I would defend

Decision 19 in the log: withholding a value means withholding every function of it.

A pre-landing review broke my central claim three ways without touching the ceiling check. Interpolating counters from floor to ceiling made every counter an exact affine function of the ceiling. Two offers the tool handed the model recovered the forbidden number to the cent. The schedule now interpolates from floor to market, both of which the model may know. "The model is never told the ceiling" was never the property. "The model cannot obtain the ceiling" is.

The honest part

The plan was: baseline 4 of 6, harden the prompt, report 6 of 6. A second baseline run on the unmodified prompt came back 6 of 6 ninety minutes later. Each defect had appeared in one run out of four. Two after-runs at zero-of-two is exactly what a fix that did nothing would produce. The defects are real. I have the transcripts and the sentence in the prompt that caused them. The effect size is unmeasured, and the artifact says so. Decision 28: a single before and a single after is not a delta.

The voice deployment

The same idea, shipped as a nine-tool, 29-node voice workflow against a TMS that speaks 1980s TCP: ASCII, CRLF-terminated, one request per connection. The protocol document and the running server disagreed in six places, each of which would have been a silent bug. Forty-three real calls were placed against the production TMS. Nine ended in a booking with a server-assigned confirmation, from $89 on a 41-mile drayage run to $7,725 on a long haul, median call 111 seconds. Three carriers were turned away at the authority gate. Bookings without an identity check: zero.

It also found a defect the 305 tests, the mutation battery, and the adversarial review had all passed over. The bundled fake TMS answered BOOKED unconditionally. The live server answers PENDING. Five consecutive real bookings reported themselves unconfirmed, and five real loads were permanently consumed. On every one of those calls the agent refused to confirm, told the carrier not to dispatch, and raised a callback. That was the correct behaviour, because at that moment the system did not know. The assertion was right. The fixture lied.

Limits, stated

• Carrier Desk is text-first. Voice was planned and deliberately not started; the review depth ate the time.
• Forty seeded lanes, one broker, one tenant. Rate ratios are global constants in a public repo. Decision 19 explains why that is survivable.
• The judge's verdicts flip on unchanged behaviour. Judged dimensions are reported, not trusted. Only code-enforced invariants gate.
• The voice deployment is a single instance, unproven above one concurrent call.

Numbers

Built with an AI coding agent doing most of the typing. The decision log, the architecture, the test bar ("must pass offline, on a plane, with the government API down"), the personas, and every pre-landing review were mine. Three of those reviews found real defects in my own central claim, and the log records each one before a reader has to find it.