Real World VoiceEQ Introduced: A New Benchmark for Evaluating Voice AI Quality

The team behind Hume AI has introduced the Real World VoiceEQ benchmark, designed to evaluate the quality of voice AI from a human perspective. It assesses over 40 leading proprietary and open-source voice AI models across 15+ key dimensions and 60+ metrics, covering automatic speech recognition (ASR), text-to-speech (TTS), speech-to-speech (S2S), and speech understanding. The benchmark is based on more than 1 million individual human evaluations, collected across diverse demographic groups, speech styles, and acoustic environments. The current version includes 785,000 TTS evaluations and 48,000 S2S evaluations, making it one of the largest human-in-the-loop studies of voice AI quality. Evaluations were conducted on the Kairos platform. According to the results, no single system ranks in the top five across all eight capability groups, highlighting that there is no single “best” voice model. Speech-to-speech models showed the greatest variance: some performed well at recognizing emotions but could not respond naturally. It turned out that access to audio does not guarantee that agents use paralinguistic information—many systems remain reliant on transcripts, ignoring tone, pauses, hesitations, intonation, and volume. The benchmark also revealed that models perform worse with accented speech, overlapping voices, emotions, background noise, and long dialogues. Word error rates for noisy speech were approximately four times higher than for music background. Additionally, some models appear to be optimized for standard benchmarks, reproducing known errors from reference transcripts. A comparison of leading speech language models (SLMs) with trained human evaluators showed that agreement is highest on tasks with clear correct answers but decreases on subjective evaluations—for example, matching a voice to an acting role or preserving identity. Automated evaluators are not yet able to replace humans when judgment depends on acoustic context and social interpretation.