GuardRate: Building an Independent Arena for Guardrail Models (Part 1)
Meta
NVIDIA
OpenAI
AllenAI
Microsoft
The HiveTrace team introduces GuardRate, a tool and leaderboard for objectively evaluating guardrail models. The tool automates benchmarking, providing transparent metrics like FPR and FNR, and highlights insights such as architecture over size. The leaderboard is now public, with the CLI source code planned for Q3 2026.
The HiveTrace team created GuardRate, a CLI tool and public leaderboard (HiveTrace GuardRate Leaderboard) to objectively compare guardrail models for LLMs. The tool automates the evaluation pipeline: it pulls datasets and model configs from YAML, spins up isolated environments per model, runs fixed benchmarks, and computes metrics (FPR, FNR, Integral Score). Results are reproducible via artifacts. The leaderboard ranks models by Integral Score; leading models include YuFeng-XGuard-Reason-8B (0.761), HiveTraceGuard-Pro 0.6B (0.743), and OpenGuardrails-Text-2510 (0.738). Insights show that architecture matters more than size, small models can beat larger ones, and latency varies widely (e.g., Llama-3.1-Nemotron-Safety-Guard-8B has p95=398.3 ms). The process reduced audit time from ~5 hours to ~30 minutes, cutting labor cost by 90%. The source code is planned for release in Q3 2026.
- Abbreviations
- LLM = Large Language Model — Большая языковая модель
- CLI = Command-Line Interface — Интерфейс командной строки
- CI/CD = Continuous Integration / Continuous Delivery — Непрерывная интеграция / непрерывная доставка
- YAML = YAML Ain't Markup Language — YAML (язык разметки)
- F1 = F1 score — F1-мера
- FPR = False Positive Rate — Доля ложных срабатываний
- FNR = False Negative Rate — Доля пропущенных угроз
- A100 = NVIDIA A100 GPU — Графический процессор NVIDIA A100
- VPS = Virtual Private Server — Виртуальный частный сервер
Source: Habr — хаб ИИ —
original