PVAD: How to Teach AI to Hear the Right Person in a Noisy Room
The Singularis team developed PVAD, a technology for detecting when a specific person speaks in a general audio stream, aiming to isolate their voice while suppressing background noise and other voices. They ran over 100 experiments, achieving target metrics for voice activity detection, but could not replicate SOTA denoising results, highlighting the limits of the approach.
The Singularis team worked on PVAD (Personal Voice Activity Detection), a technology to determine when a specific person speaks in an audio stream. Based on a short voice sample of the target speaker, the system had to identify speech segments, suppress background noise and other voices, and also attempt to replicate a state-of-the-art denoising approach without speaker separation. The project used Demucs as a base, augmented with voice embeddings from a ResNet architecture and x-vector technology, along with Kaldi, PyTorch, and TensorFlow. Training and evaluation used large public datasets including LibriSpeech and VoxCeleb2, totaling over 1 terabyte, plus small customer datasets. More than 100 experiments were conducted, and the team achieved target metrics for PVAD in both near and far microphone conditions, improving model quality on customer test data. However, the denoiser contour could not achieve SOTA-level results within the time limit, which the team considers an important finding about the approach's applicability.
- Abbreviations
- PVAD = Personal Voice Activity Detection — персональное обнаружение голосовой активности
- R&D = Research and Development — исследования и разработки
- SOTA = State of the Art — передовой уровень
- ML = Machine Learning — машинное обучение
- ResNet = Residual Network — остаточная сеть
Source: Habr — хаб ИИ —
original