Autonomy Is a Function of Verifiability
Discussions of agent autonomy tend to frame it as a trust dial: how much do you trust the model? The literature — maximalists and skeptics alike — points to a more useful framing: autonomy is not a property of the agent, it is a property of the task's feedback. An agent can safely act unattended exactly where a machine-checkable success signal exists and mistakes are cheap to revert. Everywhere else, it should escalate.
The math that sets the stakes
Start with the skeptic's arithmetic (Utkarsh Kanwat, a builder writing against his own book): if each step of an unattended workflow succeeds with probability 0.95, a 20-step chain succeeds 0.95²⁰ ≈ 36% of the time. Even at 99% per step you get ~82% — nowhere near production-grade. This is not a prompt-engineering problem; it's multiplication. Anthropic's own guidance says the same thing in softer language: autonomous agents compound errors and require sandboxed testing and guardrails (Building Effective Agents).
Two engineering responses follow, and you need both:
- Shorten the unattended chain — insert checkpoints where a signal (test, exit code, review) resets the error budget. A 20-step chain with a verified checkpoint every 5 steps is four 5-step chains: each at 0.95⁵ ≈ 77%, each independently retryable. Verification checkpoints are how you beat the exponent.
- Bound the blast radius — arrange the world so a wrong step lands somewhere cheap. A feature branch instead of main. A sandbox instead of prod. A draft instead of a sent message. When iteration is cheap, 36% first-pass success is fine — you just run the loop again.
Grant autonomy per action class, not per agent
Both major labs' deployment guidance converges here. OpenAI's guide names exactly two escalation triggers: exceeding a failure/retry threshold, and any high-risk action — sensitive, irreversible, or high-stakes — with human oversight of the latter retained until reliability is proven. It also prescribes per-tool risk ratings (read-only vs write, reversibility, financial impact) with layered guardrails, and reports that incremental autonomy rollout beats building the fully-autonomous version first. Anthropic's safety framework places human control before high-stakes decisions, not over every action, operationalized in Claude Code as read-only by default, mutating actions gated, with permissions granted incrementally per action class.
The operational hardware
Persistent-agent practitioners converge on a small kit:
- Circuit breakers. After N consecutive failures, stop and cool down (one production example: 3 failures → 30-minute cooldown), plus missed-run detection at startup. An agent that retries forever is a cost bug and a thrash bug — and per Part 3, a session polluted by failed attempts gets worse, not better. Stop, flag, restart fresh.
- Immutable body, mutable brain. Keep the runtime the agent cannot break — daemon, container, entrypoint, cron — separate from the state it freely edits (memory files). The practitioner who coined the split reports agents do corrupt their own instructions in the wild; the immutable body is why that's an incident, not an outage.
- Engineer away failure classes. When an unattended run fails, the response isn't "retry harder" — it's a permanent harness/tooling fix so that class of failure can't recur (Geoffrey Huntley's loop doctrine). Reliability accretes by deletion of failure modes, not accumulation of cleverness. And most failures are exactly this kind: rate limits, memory bloat, git conflicts — operational, not model intelligence.
- ~70% of the work is tools. Kanwat's production estimate: the model contributes ~30%; tool/feedback engineering — state communication, partial success signals, recovery — is the rest. Anthropic's SWE-bench team said the same from the other side: they spent more time optimizing tools than the prompt.
Applying it
Write the policy down as action classes. XiaoSteve's version, which mirrors the quadrant: writes, commits, installs and pr-branch pushes are autonomous (verifiable, revertable, branch-bounded); shared-branch pushes are forbidden outright (merges happen only through reviewed PRs); SSH/firewall changes are never touched (irreversible lockout risk); stuck-twice means stop and flag. The point isn't those specific rules — it's that "how autonomous is the agent?" has no single answer, and doesn't need one.
Next: the same logic settles the loudest fight in agent-land — one agent or many? Part 5 →