What Building the Shippy Agent Taught Us: How to Build Reliable AI Agents for Operational Tasks

The Skylight team (a division of Ai2) has described the creation of Shippy, an AI agent for real-time maritime awareness. The main challenge is reliability: a mistake could send a patrol vessel to the wrong location. Shippy's architecture consists of three parts: a soul (system prompt defining behavior boundaries), skills (a set of instructions in Markdown format with front matter), and configuration (OpenClaw execution environment, LLM Claude Opus 4.6, runtime parameters). Skills describe typical queries: searching for events, exclusive economic zone boundaries, interpreting vessel tracks, and generating map links. To reduce nondeterminism, Shippy accesses the Skylight API through a specialized CLI that handles authentication, pagination, and output formatting. This eliminates errors inherent in direct calls. To isolate user data, the Mothership platform was built: for each session, a separate Kubernetes deployment is created, tied to the user's JWT. Agent evaluation is conducted on live data: experts write scenarios with rubrics and weights, an LLM judge evaluates each criterion, results are aggregated and compared against a threshold. Tests are run via the Harbor framework. Shippy is open to early users; plans include map management through the agent, routing queries to different models, and cross-session memory. Shippy's experience is also being used for the EarthRanger and OlmoEarth projects.