What Building Shippy Taught Us About Building Reliable AI Agents for Operational Tasks

The Skylight team at Ai2 built Shippy, an AI agent for real-time maritime domain awareness, focusing on reliability. Shippy is composed of three parts: soul (system prompt), skills (markdown files specifying tasks), and config (runtime settings). Skills are plain markdown files following the agent-skills spec used by Claude Code and Codex. Instead of calling the API directly, Shippy uses a purpose-built CLI (Skylight CLI) that handles authentication, pagination, and structured output. User sessions are isolated via Mothership, which provisions a dedicated Kubernetes deployment per session. The evaluation system uses subject-matter experts to write scenarios and rubrics, scoring the agent on live data with an LLM judge. Shippy currently uses Claude Opus 4.6 as its LLM. Future plans include agent-driven UI control, model routing, and cross-thread memory.