Speech-to-Text Translation

Translate audio signals of speech in one language into text in a foreign language, either in an end-to-end or cascade manner.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 26–50 of 146 papers

Title	Date	Tasks	Status	Hype
Investigating Decoder-only Large Language Models for Speech-to-text Translation	Jul 3, 2024	Decoderparameter-efficient fine-tuning	—Unverified	0
NAIST Simultaneous Speech Translation System for IWSLT 2024	Jun 30, 2024	Speech-to-Speech TranslationSpeech-to-Text	—Unverified	0
Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects	Jun 27, 2024	Automatic Speech RecognitionMachine Translation	CodeCode Available	0
ArzEn-LLM: Code-Switched Egyptian Arabic-English Translation and Speech Recognition Using LLMs	Jun 26, 2024	ArzEn Code-switched Translation to araArzEn Code-switched Translation to eng	CodeCode Available	1
SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation	Jun 20, 2024	Speech-to-TextSpeech-to-Text Translation	CodeCode Available	0
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?	Jun 11, 2024	Contrastive LearningSpeech Synthesis	—Unverified	0
StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection	Jun 10, 2024	Speech-to-TextSpeech-to-Text Translation	CodeCode Available	0
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning	Jun 5, 2024	Automatic Speech Recognition (ASR)de-en	CodeCode Available	5
LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions	May 18, 2024	Language ModelingLanguage Modelling	CodeCode Available	1
Robust Semantic Communications for Speech Transmission	Mar 8, 2024	Generative Adversarial NetworkSemantic Communication	—Unverified	0
Compact Speech Translation Models via Discrete Speech Units Pretraining	Feb 29, 2024	DecoderSelf-Supervised Learning	—Unverified	0
Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?	Feb 19, 2024	Speech-to-TextSpeech-to-Text Translation	—Unverified	0
Pushing the Limits of Zero-shot End-to-End Speech Translation	Feb 16, 2024	Speech-to-TextSpeech-to-Text Translation	CodeCode Available	1
Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases	Feb 1, 2024	speech-recognitionSpeech Recognition	—Unverified	0
Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision	Dec 30, 2023	Speech-to-TextSpeech-to-Text Translation	CodeCode Available	0
Efficient Monotonic Multihead Attention	Dec 7, 2023	Simultaneous Speech-to-Text TranslationSpeech-to-Text	—Unverified	0
End-to-End Speech-to-Text Translation: A Survey	Dec 2, 2023	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning	Nov 3, 2023	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation	Nov 1, 2023	Automatic Speech Recognitionspeech-recognition	CodeCode Available	1
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT	Oct 7, 2023	Audio captioningAutomatic Speech Recognition	CodeCode Available	2
Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach	Oct 6, 2023	Simultaneous Speech-to-Text TranslationSpeech-to-Text	—Unverified	0
Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer	Oct 5, 2023	Speech-to-TextSpeech-to-Text Translation	—Unverified	0
Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing	Sep 27, 2023	DecoderMachine Translation	—Unverified	0
SpeechAlign: a Framework for Speech Translation Alignment Evaluation	Sep 20, 2023	Speech-to-TextSpeech-to-Text Translation	—Unverified	0
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders	Sep 14, 2023	Contrastive LearningKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 2 of 6Next →

All datasets MuST-C EN->DE MuST-C EN->ES MuST-C EN->FR CoVoST 2 eng-X CoVoST 2 X-eng FLEURS eng-X FLEURS X-eng libri-trans MediBeng MuST-C MuST-C EN->NL

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Task Modulation + Multitask Learning(ASR/MT) + Data Augmentation	Case-sensitive sacreBLEU	28.88	—	Unverified
2	Wav2Vec2.0+mBART+Adaptors	Case-sensitive sacreBLEU	28.22	—	Unverified
3	Transformer + Meta Learning(ASR/MT) + Data Augmentation	Case-sensitive sacreBLEU	27.51	—	Unverified
4	Transformer with Adapters	Case-sensitive sacreBLEU	24.63	—	Unverified
5	Dual-decoder Transformer	Case-sensitive sacreBLEU	23.63	—	Unverified
6	Speechformer	Case-sensitive sacreBLEU	23.6	—	Unverified
7	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	22.8	—	Unverified
8	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	22.7	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Transformer with Adapters	Case-sensitive sacreBLEU	28.73	—	Unverified
2	Speechformer	Case-sensitive sacreBLEU	28.5	—	Unverified
3	Dual-decoder Transformer	Case-sensitive sacreBLEU	28.12	—	Unverified
4	Transformer + ASR Pretrain + SpecAug	Case-sensitive sacreBLEU	27.4	—	Unverified
5	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	26.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Dual-decoder Transformer	Case-sensitive sacreBLEU	33.45	—	Unverified
2	Transformer + ASR Pretrain + SpecAug	Case-sensitive sacreBLEU	33.3	—	Unverified
3	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	32.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	30.6	—	Unverified
2	SeamlessM4T Medium	BLEU	26.6	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	34.1	—	Unverified
2	SeamlessM4T Medium	BLEU	29.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	21.5	—	Unverified
2	SeamlessM4T Medium	BLEU	19.2	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	24	—	Unverified
2	SeamlessM4T Medium	BLEU	20.9	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Transformer + ASR Pretrain + SpecAug	Case-insensitive sacreBLEU	17.2	—	Unverified
2	Transformer + ASR Pretrain	Case-insensitive sacreBLEU	16.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	MediBeng Whisper Tiny	Bleu	0.98	—	Unverified
2	Whisper Tiny	Bleu	0.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Transformer with Adapters	SacreBLEU	26.61	—	Unverified
2	Dual-decoder Transformer	SacreBLEU	25.62	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Speechformer	Case-sensitive sacreBLEU	27.7	—	Unverified