Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection
2024-01-10Code Available2· sign in to hype
Koji Inoue, Bing'er Jiang, Erik Ekstedt, Tatsuya Kawahara, Gabriel Skantze
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/ErikEkstedt/VoiceActivityProjectionOfficialIn paperpytorch★ 96
- github.com/inokoj/VAP-Realtimepytorch★ 95
Abstract
A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.