Three AI Incidents Caused by Lack of Architecture: Deloitte, Flock, and a $38 MRR SaaS Disaster
Anthropic
OpenAI
Analysis of three AI incidents where problems stemmed from poor system architecture rather than model errors. Deloitte Australia's report AI invented sources; Flock's license plate system wrongly flagged a journalist; a SaaS founder discovered an AI agent canceled all subscriptions overnight, dropping MRR to $38. Solutions involve deterministic checks, validations, and human-in-the-loop mechanisms.
An architect of LLM systems analyzed three high-profile AI incidents, each caused by different architectural flaws: Deloitte Australia's AI-generated report contained fabricated sources despite human reviews, Flock's camera network misidentified a journalist's car as stolen due to a truncated plate number, and a SaaS founder found that an AI agent written by a model canceled all subscriptions, reducing MRR to $38. The common root was that people who know prompt engineering but lack system design built the architectures. The author proposes three principles: control must live in deterministic code, not prompts; probabilistic components should not replace deterministic ones in critical paths; and strictness of controls must be calibrated to the cost of error. Solutions include using citations with mechanical verification, schema validation for structured data, and mandatory human-in-the-loop for high-risk actions.
- Abbreviations
- LLM = Large Language Model
- MRR = Monthly Recurring Revenue
- API = Application Programming Interface
- SaaS = Software as a Service
Source: Habr — хаб ИИ —
original