SOTAVerified|Agents Browse Leaderboard About

Audio-Visual Speech Recognition

Audio-visual speech recognition is the task of transcribing a paired audio and visual stream into text.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 51–60 of 100 papers

Title	Date	Tasks	Status	Hype
Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices	Feb 17, 2023	Audio-Visual Speech RecognitionGesture Recognition	—Unverified	0
AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations	Feb 10, 2023	Audio-Visual Speech RecognitionSelf-Supervised Learning	—Unverified	0
A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset	Jan 21, 2023	Audio-Visual Speech RecognitionAutomatic Speech Recognition	—Unverified	0
OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset	Jan 16, 2023	Audio-Visual Speech RecognitionLip Reading	CodeCode Available	1
ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration	Jan 1, 2023	Audio-Visual Speech RecognitionResynthesis	—Unverified	0
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement	Dec 21, 2022	Audio-Visual Speech RecognitionResynthesis	—Unverified	0
Jointly Learning Visual and Auditory Speech Representations from Raw Data	Dec 12, 2022	Audio-Visual Speech RecognitionLipreading	CodeCode Available	1
Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning	Dec 10, 2022	Audio-Visual Speech Recognitionreinforcement-learning	—Unverified	0
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning	Nov 21, 2022	Audio-Visual Speech RecognitionLanguage Modelling	—Unverified	0
Streaming Audio-Visual Speech Recognition with Alignment Regularization	Nov 3, 2022	Audio-Visual Speech RecognitionAutomatic Speech Recognition	—Unverified	0

Show:10 25 50

← PrevPage 6 of 10Next →

All datasets LRS2 LRS3-TED LRW CAS-VSR-S101

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Hybrid CTC / Attention	Word Error Rate (WER)	39.1	—	Unverified
2	TM-Seq2seq	Test WER	8.5	—	Unverified
3	TM-CTC	Test WER	8.2	—	Unverified
4	CTC/Attention	Test WER	7	—	Unverified
5	CTC/Attention	Test WER	1.5	—	Unverified
6	Whisper-Flamingo	Test WER	1.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Hyb-Conformer	Word Error Rate (WER)	2.3	—	Unverified
2	Zero-AVSR	Word Error Rate (WER)	1.5	—	Unverified
3	AV-HuBERT Large	Word Error Rate (WER)	1.4	—	Unverified
4	Whisper-Flamingo	Word Error Rate (WER)	0.76	—	Unverified
5	MMS-LLaMA	Word Error Rate (WER)	0.74	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	AVCRFormer	Top-1 Accuracy	98.81	—	Unverified
2	2DCNN + BiLSTM + ResNet + MLF	Top-1 Accuracy	98.76	—	Unverified
3	PBL	Top-1 Accuracy	98.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ES³ Base*	Word Error Rate (WER)	11	—	Unverified