Rails Testing on Autopilot: AI Agent That Writes Tests Instead of Developers
Mistral
Mistral AI has created an autonomous agent based on the open-source tool Vibe that automatically generates and improves RSpec tests for Rails monoliths. The agent reads source files, creates or refines tests, checks their style with RuboCop, runs them through RSpec with SimpleCov, and fixes errors. In an experiment with 275 files, coverage reached 100%, all tests passed, and the average LLM-as-a-judge score was 0.74.
Mistral AI has developed an autonomous agent for testing Rails applications based on its open-source tool Vibe. The agent reads Rails source files (models, serializers, controllers, mailers, helpers) and generates or improves RSpec tests, following file-matching conventions. Separate skill files are used for different file types. The agent runs RuboCop for style checking and SimpleCov together with RSpec for coverage and correctness verification — this is a mandatory final step, since without execution, tests may appear high-quality but fail due to syntax errors. The AGENTS.md file describes a step-by-step workflow including enforced self-check: the agent rereads the source file and verifies that all public methods are tested. On the first run, only one-third of the tests passed, but the agent fixed all errors within a few iterations. In an experiment with 275 files (half without tests), the agent generated or improved tests, achieving 100% test passing, zero RuboCop violations, 100% SimpleCov coverage, and an average score of 0.74 using the LLM-as-a-judge method. Models performed best (0.81), while controllers performed worst (0.67).
Source: Mistral AI —
original