416 tests and a 'destroy all' button: where agent projects break
Anthropic
An analysis of six agent projects reveals recurring flaws: safety as text, irreversible actions without gates, and zero behavioral tests. Even a mature open-source project with regression tests still lacks safeguards for irreversible email sending. The author proposes ten diagnostic questions to assess agent system maturity.
The author has been running an autonomous agent called Groundhog (Surok) since February 2026 on a research loop with Claude Code and --dangerously-skip-permissions. They then audited six of their own agent projects against a composite checklist from six sources (including Sber's AI-DISRUPT PDLC methodology) and found three recurring failure patterns: safety enforced only in prompts (model can be overridden), irreversible actions without confirmation (e.g., a rebuild command wipes all enriched data), and zero regression tests for behavioral bugs. The author also reviewed an anonymous large open-source agent project: it had code-level permission checks and regression tests for past bugs, but still sent emails irreversibly without a draft/confirmation step. A newer open-source framework (HarnessX) offers an interrupt_on gate but it's optional and not default. Sber's PDLC methodology formalizes many of the missing controls: policy as code, evals as completion contracts, and an R0–R5 permission ladder with mandatory abort/rollback for state-changing operations at R3+. The author built a seventh project implementing some lessons learned: now evals run before each release and incidents become named regression tests.
- Сокращения
- PDLC = Product Development Life Cycle — жизненный цикл разработки продукта
- MCP = Model Context Protocol — протокол контекста модели
- SMTP = Simple Mail Transfer Protocol — простой протокол передачи почты
- cron = cron (command scheduler) — планировщик задач cron
Source: Habr — хаб ИИ —
original
