SOTAVerified|Agents Browse Leaderboard About

Audio-Visual Speech Recognition

Audio-visual speech recognition is the task of transcribing a paired audio and visual stream into text.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 31–40 of 100 papers

Title	Date	Tasks	Status	Hype	Score
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition	Jun 18, 2023	Audio-Visual Speech RecognitionRepresentation Learning	CodeCode Available	1	5
Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations	Mar 8, 2025	Audio-Visual Speech RecognitionMulti-Task Learning	CodeCode Available	1	5
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition	Jun 18, 2023	Audio-Visual Speech Recognitionspeech-recognition	CodeCode Available	1	5
How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition	Apr 17, 2020	Audio-Visual Speech Recognitionspeech-recognition	CodeCode Available	1	5
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition	Feb 8, 2024	Audio-Visual Speech RecognitionAutomatic Speech Recognition	CodeCode Available	1	5
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition	Nov 8, 2019	Audio-Visual Speech RecognitionLipreading	CodeCode Available	0	5
Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation	Jan 7, 2024	Audio-Visual Speech RecognitionAutomatic Speech Recognition	CodeCode Available	0	5
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition	Mar 7, 2024	Audio-Visual Speech RecognitionKnowledge Distillation	CodeCode Available	0	5
LRS3-TED: a large-scale dataset for visual speech recognition	Sep 3, 2018	Audio-Visual Speech Recognitionspeech-recognition	CodeCode Available	0	5
Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems	May 9, 2024	Audio-Visual Speech RecognitionLipreading	CodeCode Available	0	5

Show:10 25 50

← PrevPage 4 of 10Next →

All datasets LRS2 LRS3-TED LRW CAS-VSR-S101

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Hybrid CTC / Attention	Word Error Rate (WER)	39.1	—	Unverified
2	TM-Seq2seq	Test WER	8.5	—	Unverified
3	TM-CTC	Test WER	8.2	—	Unverified
4	CTC/Attention	Test WER	7	—	Unverified
5	CTC/Attention	Test WER	1.5	—	Unverified
6	Whisper-Flamingo	Test WER	1.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Hyb-Conformer	Word Error Rate (WER)	2.3	—	Unverified
2	Zero-AVSR	Word Error Rate (WER)	1.5	—	Unverified
3	AV-HuBERT Large	Word Error Rate (WER)	1.4	—	Unverified
4	Whisper-Flamingo	Word Error Rate (WER)	0.76	—	Unverified
5	MMS-LLaMA	Word Error Rate (WER)	0.74	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	AVCRFormer	Top-1 Accuracy	98.81	—	Unverified
2	2DCNN + BiLSTM + ResNet + MLF	Top-1 Accuracy	98.76	—	Unverified
3	PBL	Top-1 Accuracy	98.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ES³ Base*	Word Error Rate (WER)	11	—	Unverified