How to train an infrastructure LLM agent not to lie
DeepSeek
A read-only AI agent for infrastructure incident investigations was built. Initially it gave plausible but useless answers. The team solved the problem by moving LLM inside the application, adding tool contracts, evidence guards, and structured planning. In the last run, 44 of 45 test scenarios passed.
An internal read-only LLM agent was built to answer infrastructure questions like 'why is the service returning 502?'. It used Go, without LangChain, and called models DeepSeek-V4-Flash and Qwen3.6-27B-FP8 via OpenAI-compatible API. The first version used a free ReAct loop and produced answers that were formally correct but often misleading. For example, when a 502 error was caused by an oversized response header at the proxy, the agent concluded 'no error at the application level', missing the real cause. Five classes of errors were identified: wrong scope, wrong time semantics, wrong negative conclusions, loss of evidence, and uncontrolled replanning. The solution was to embed the LLM inside a strict application framework. Tool calls are governed by an extended contract x-agent that defines evidence, output kind, freshness, negative semantics, and read-only status. The planner first builds objectives (e.g., symptom.http_502, route.edge_to_ingress) and a stop condition: every required objective must be resolved or explicitly unavailable. An evidence guard prevents the finalizer from concluding before all objectives are met. Tool selection is staged: policy filter, lexical prefilter, LLM rerank (top 8), then objective relevance gate. Risky intents have deterministic pre-routes. In the final evaluation, 44 out of 45 scenarios passed (29 of 29 mandatory), running twice each. The only failed test was an optional capacity report format. The agent remains read-only; humans decide on production changes.
- Сокращения
- LLM = Large Language Model — Большая языковая модель
- MCP = Model Context Protocol — Протокол контекста модели
- SRE = Site Reliability Engineering — Инжиниринг надёжности сайтов
Source: Habr — хаб ИИ —
original
