Speech-to-Text Translation

Translate audio signals of speech in one language into text in a foreign language, either in an end-to-end or cascade manner.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 51–75 of 146 papers

Title	Date	Tasks	Status	Hype
An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation	Aug 28, 2023	Machine TranslationNMT	CodeCode Available	0
SONAR: Sentence-Level Multimodal and Language-Agnostic Representations	Aug 22, 2023	DecoderMachine Translation	CodeCode Available	2
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation	Aug 22, 2023	Automatic Speech RecognitionMachine Translation	CodeCode Available	2
On decoder-only architecture for speech-to-text and large language model integration	Jul 8, 2023	DecoderLanguage Modeling	—Unverified	0
AudioPaLM: A Large Language Model That Can Speak and Listen	Jun 22, 2023	Language ModelingLanguage Modelling	—Unverified	0
Recent Advances in Direct Speech-to-text Translation	Jun 20, 2023	Data AugmentationDecoder	—Unverified	0
Improved Cross-Lingual Transfer Learning For Automatic Speech Translation	Jun 1, 2023	automatic-speech-translationCross-Lingual Transfer	—Unverified	0
Strategies for improving low resource speech to text translation relying on pre-trained ASR models	May 31, 2023	Automatic Speech RecognitionDecoder	—Unverified	0
ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation	May 24, 2023	GPULanguage Modeling	CodeCode Available	1
DUB: Discrete Unit Back-translation for Speech Translation	May 19, 2023	Machine TranslationSpeech-to-Text	CodeCode Available	1
Back Translation for Speech-to-text Translation Without Transcripts	May 15, 2023	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	CodeCode Available	1
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks	May 4, 2023	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit	Apr 10, 2023	BenchmarkingSimultaneous Speech-to-Text Translation	—Unverified	0
Enhancing Speech-to-Speech Translation with Multiple TTS Targets	Apr 10, 2023	Speech-to-Speech TranslationSpeech-to-Text	—Unverified	0
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages	Mar 2, 2023	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation	Mar 1, 2023	Audio-Visual Speech RecognitionRobust Speech Recognition	CodeCode Available	2
Pre-training for Speech Translation: CTC Meets Optimal Transport	Jan 27, 2023	Multi-Task LearningSpeech-to-Text	CodeCode Available	1
WACO: Word-Aligned Contrastive Learning for Speech Translation	Dec 19, 2022	Contrastive LearningSpeech-to-Text	CodeCode Available	0
M3ST: Mix at Three Levels for Speech Translation	Dec 7, 2022	Data AugmentationDiversity	—Unverified	0
Efficient Speech Translation with Dynamic Latent Perceivers	Oct 28, 2022	Speech-to-TextSpeech-to-Text Translation	CodeCode Available	0
Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translation	Oct 24, 2022	SegmentationSpeech-to-Text	CodeCode Available	0
Simple and Effective Unsupervised Speech Translation	Oct 18, 2022	Domain AdaptationMachine Translation	—Unverified	0
CTC Alignments Improve Autoregressive Translation	Oct 11, 2022	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation	Jul 3, 2022	DecoderSpeech-to-Text	CodeCode Available	0
Language Model Augmented Monotonic Attention for Simultaneous Translation	Jul 1, 2022	Language ModelingLanguage Modelling	—Unverified	0

Show:10 25 50

← PrevPage 3 of 6Next →

All datasets MuST-C EN->DE MuST-C EN->ES MuST-C EN->FR CoVoST 2 eng-X CoVoST 2 X-eng FLEURS eng-X FLEURS X-eng libri-trans MediBeng MuST-C MuST-C EN->NL

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Task Modulation + Multitask Learning(ASR/MT) + Data Augmentation	Case-sensitive sacreBLEU	28.88	—	Unverified
2	Wav2Vec2.0+mBART+Adaptors	Case-sensitive sacreBLEU	28.22	—	Unverified
3	Transformer + Meta Learning(ASR/MT) + Data Augmentation	Case-sensitive sacreBLEU	27.51	—	Unverified
4	Transformer with Adapters	Case-sensitive sacreBLEU	24.63	—	Unverified
5	Dual-decoder Transformer	Case-sensitive sacreBLEU	23.63	—	Unverified
6	Speechformer	Case-sensitive sacreBLEU	23.6	—	Unverified
7	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	22.8	—	Unverified
8	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	22.7	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Transformer with Adapters	Case-sensitive sacreBLEU	28.73	—	Unverified
2	Speechformer	Case-sensitive sacreBLEU	28.5	—	Unverified
3	Dual-decoder Transformer	Case-sensitive sacreBLEU	28.12	—	Unverified
4	Transformer + ASR Pretrain + SpecAug	Case-sensitive sacreBLEU	27.4	—	Unverified
5	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	26.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Dual-decoder Transformer	Case-sensitive sacreBLEU	33.45	—	Unverified
2	Transformer + ASR Pretrain + SpecAug	Case-sensitive sacreBLEU	33.3	—	Unverified
3	Transformer + ASR Pretrain	Case-sensitive sacreBLEU	32.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	30.6	—	Unverified
2	SeamlessM4T Medium	BLEU	26.6	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	34.1	—	Unverified
2	SeamlessM4T Medium	BLEU	29.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	21.5	—	Unverified
2	SeamlessM4T Medium	BLEU	19.2	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SeamlessM4T Large	BLEU	24	—	Unverified
2	SeamlessM4T Medium	BLEU	20.9	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Transformer + ASR Pretrain + SpecAug	Case-insensitive sacreBLEU	17.2	—	Unverified
2	Transformer + ASR Pretrain	Case-insensitive sacreBLEU	16.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	MediBeng Whisper Tiny	Bleu	0.98	—	Unverified
2	Whisper Tiny	Bleu	0.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Transformer with Adapters	SacreBLEU	26.61	—	Unverified
2	Dual-decoder Transformer	SacreBLEU	25.62	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Speechformer	Case-sensitive sacreBLEU	27.7	—	Unverified