SOTAVerified

Audio-Visual Speech Recognition

Audio-visual speech recognition is the task of transcribing a paired audio and visual stream into text.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–25 of 100 papers

TitleStatusHype
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition—0
Cocktail-Party Audio-Visual Speech Recognition—0
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach—0
The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition—0
SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer—0
CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative SynchronizationCode2
Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides—0
Visual-Aware Speech Recognition for Noisy Scenarios—0
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech TokensCode1
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs—0
Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech RepresentationsCode1
MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition—0
Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation ModelsCode1
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech RecognitionCode3
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models—0
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech RepresentationCode1
Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech RecognitionCode0
Uncovering the Visual Contribution in Audio-Visual Speech Recognition—0
Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective—0
Large Language Models are Strong Audio-Visual Speech Recognition LearnersCode2
DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module—0
SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic DataCode0
Tailored Design of Audio-Visual Speech Recognition Models using BranchformersCode1
Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech RecognitionCode1
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization—0
Show:102550
← PrevPage 1 of 4Next →
All datasetsLRS2LRS3-TEDLRWCAS-VSR-S101

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Hybrid CTC / AttentionWord Error Rate (WER)39.1—Unverified
2TM-Seq2seqTest WER8.5—Unverified
3TM-CTCTest WER8.2—Unverified
4CTC/AttentionTest WER7—Unverified
5CTC/AttentionTest WER1.5—Unverified
6Whisper-FlamingoTest WER1.4—Unverified
#ModelMetricClaimedVerifiedStatus
1Hyb-ConformerWord Error Rate (WER)2.3—Unverified
2Zero-AVSRWord Error Rate (WER)1.5—Unverified
3AV-HuBERT LargeWord Error Rate (WER)1.4—Unverified
4Whisper-FlamingoWord Error Rate (WER)0.76—Unverified
5MMS-LLaMAWord Error Rate (WER)0.74—Unverified
#ModelMetricClaimedVerifiedStatus
1AVCRFormerTop-1 Accuracy98.81—Unverified
22DCNN + BiLSTM + ResNet + MLFTop-1 Accuracy98.76—Unverified
3PBLTop-1 Accuracy98.3—Unverified
#ModelMetricClaimedVerifiedStatus
1ES³ Base*Word Error Rate (WER)11—Unverified