Making an LLM safe enough to stand next to dangerous work

I spent a few months building a voice-and-camera agent that coaches HVAC technicians through physical tasks. You point a phone at the equipment, it sees what you see, and it talks you through the next step. Replacing a capacitor. Pulling a disconnect. The hard problem was never the language model. The hard problem was that some of the steps can hurt you, and a model that is a little too agreeable will tell you it's safe to proceed when it isn't.

This is the design that made the agent trustworthy enough to stand next to that work. It is a design and not a code drop, because the code belongs to the company I built it at. But the shape is worth sharing. It is the same shape as every high-stakes agent problem I've seen since.

The failure you are designing against

On a capacitor job there is one step that matters more than the rest: discharge it before you touch it. A charged capacitor can injure someone. So the workflow must not advance past that step until the technician confirms they discharged it, and it must not be trickable into advancing.

The naive build asks the model "did they say they discharged it?" and advances on yes. That fails in two directions. It fails open when someone says "discharged?" as a question, or "almost discharged," and the model, wanting to be helpful, hears a confirmation that isn't there. And it fails silently, because a model that hallucinates a yes leaves no trace that looks any different from a real one.

Cheap and certain first, expensive and smart last

The architecture is four layers, ordered so the model is reached only for what actually needs judgment.

A deterministic state machine owns where you are in the workflow and every hard gate. A keyword fast path handles the common utterances, "next step," "check my work," before any model call. A small, cheap classifier decides intent when the words are less obvious, and it returns "do nothing" on any error instead of raising. Only then does the coaching model run. And even it can't advance the workflow on its own. Any advance it proposes gets downgraded to "stay" unless a human explicitly asked to move on, by voice or by button. That rule is marked in the codebase as load-bearing and never to be removed, because the whole risk is a model that helpfully completes a step nobody finished.

The gate is an AND, and it fails closed

The safety gate is the centerpiece, and on purpose it is not one model call.

The floor is pure code. A whole-token match on the keyword ("discharged"), plus a scan of the whole utterance for negation. No model. It is the layer you can prove correct, and it is frozen, byte for byte unchanged by policy, because the thing you can prove is the thing you don't let drift.

The floor can't catch everything. "Discharged?" and "almost discharged" both contain the keyword and no negation word, so on its own the floor would open the gate on a question and on a hedge. So a second, narrow model call answers exactly one thing: did they confidently say they already did it? Nothing else. High confidence threshold.

Both must agree to open the gate. It is an AND, not an OR. And every error path in both layers fails closed, toward the gate staying shut. The property that lets me sleep is that the composition only degrades in one direction. A judge that is getting worse can only become more likely to keep the gate closed, which is annoying, and never more likely to open it, which is dangerous. That asymmetry is what let me ship without a full nightly recall harness for the judge and defend the decision in writing. The worst a broken judge can do is make a technician repeat themselves.

Every hardening pass came from an attack that worked

I didn't design the gate and declare it safe. I attacked it, and each bypass that worked became a fix and a permanent test. There were five. They live now as a corpus of adversarial phrases with monotonicity tests in continuous integration, so a regression that would let a wrong judge open a denied gate fails the build.

The corpus is bilingual, English and Russian, and that is the part I'm proudest of, because it is the proof that the design generalizes instead of being a pile of English string hacks. The negation floor takes a pluggable tokenizer and a negation set. A Russian denial, "I haven't discharged it," is caught by the same whole-utterance scan, just with the Russian negation particle. Same mechanism, second language, no new special cases.

What I will tell you is not done

You can tell whether someone built a thing or narrated it by whether they can name what's missing. Here is what was missing.

The golden evals are all-must-pass with no percentage thresholds. A nightly recall-tracking harness for the judge was designed and deliberately deferred, for the fail-closed reason above. The vision test fixtures were placeholder images, so the pipeline was tested but real visual recognition was not. Token and cache metrics were computed but not emitted to traces for a while, which means a cost regression would have been invisible. And camera frames had no staleness timestamp for a stretch, so the model could reason about a thirty-second-old frame without knowing it was old.

The transferable part

Strip out the HVAC and what's left is the shape of every serious agent. The thing that must not happen is enforced by code that can't express the violation. The model is reached only for judgment and is never the last word on a safety-critical transition. The certain layer is frozen, and the smart layer is allowed to be imperfect, because the composition is safe either way. And the whole thing is hardened by attacks that actually worked, not attacks you imagined.

The stakes here were literal: a person with their hands inside a machine. That only made the discipline unavoidable. It was always the right discipline.