SOTAVerified

Speech-to-Speech Translation

Speech-to-speech translation (S2ST) consists on translating speech from one language to speech in another language. This can be done with a cascade of automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis sub-systems, which is text-centric. Recently, works on S2ST without relying on intermediate text representation is emerging.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 51–100 of 117 papers

TitleStatusHype
SimulTron: On-Device Simultaneous Speech to Speech Translation—0
Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing—0
SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought—0
CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning—0
DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech TranslationCode0
MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation—0
Direct Punjabi to English speech translation using discrete units—0
A Case Study on Filtering for End-to-End Speech Translation—0
TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data—0
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation—0
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation—0
Enhancing expressivity transfer in textless speech-to-speech translation—0
Direct Text to Speech Translation System using Acoustic Units—0
Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer—0
Multilingual Speech-to-Speech Translation into Multiple Target Languages—0
Towards cross-language prosody transfer for dialogCode0
AudioPaLM: A Large Language Model That Can Speak and Listen—0
PolyVoice: Language Models for Speech to Speech Translation—0
Translatotron 3: Speech to Speech Translation with Monolingual Data—0
Textless Speech-to-Speech Translation With Limited Parallel DataCode0
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation—0
i-Code Studio: A Configurable and Composable Framework for Integrative AI—0
Duplex Diffusion Models Improve Speech-to-Speech Translation—0
ESPnet-ST-v2: Multipurpose Spoken Language Translation ToolkitCode0
Enhancing Speech-to-Speech Translation with Multiple TTS Targets—0
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation—0
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete UnitsCode0
Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features—0
Dialogs Re-enacted Across LanguagesCode0
Speech-to-Speech Translation For A Real-world Unwritten Language—0
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations—0
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech TranslationCode0
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation—0
Improving Speech-to-Speech Translation Through Unlabeled Text—0
A Textless Metric for Speech-to-Speech ComparisonCode0
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling—0
SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation—0
Findings of the IWSLT 2022 Evaluation Campaign—0
Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022Code0
The HW-TSC’s Speech to Speech Translation System for IWSLT 2022 Evaluation—0
MLLP-VRAIN UPV systems for the IWSLT 2022 Simultaneous Speech Translation and Speech-to-Speech Translation tasks—0
LibriS2S: A German-English Speech-to-Speech Translation CorpusCode0
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation—0
Prosodic Alignment for off-screen automatic dubbing—0
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation—0
Evaluating MT Systems: A Theoretical Framework—0
Textless Speech-to-Speech Translation on Real Data—0
Multimodal and Multilingual Embeddings for Large-Scale Speech MiningCode0
Assessing Evaluation Metrics for Speech-to-Speech Translation—0
From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation—0
Show:102550
← PrevPage 2 of 3Next →
All datasetsTATFLEURS X-engCVSS

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Hokkien→En (Two-pass decoding)ASR-BLEU (Dev)13.6—Unverified
2Hokkien→En (Three-stage)ASR-BLEU (Dev)12.5—Unverified
3Hokkien→En (Two-stage)ASR-BLEU (Dev)12.5—Unverified
4Hokkien→En (Single-pass decoding)ASR-BLEU (Dev)8.8—Unverified
5En→Hokkien (Two-pass decoding)ASR-BLEU (Dev)7.8—Unverified
6En→Hokkien (Three-stage)ASR-BLEU (Dev)7.5—Unverified
7En→Hokkien (Two-stage)ASR-BLEU (Dev)7.1—Unverified
8En→Hokkien (Single-pass decoding)ASR-BLEU (Dev)6.6—Unverified
#ModelMetricClaimedVerifiedStatus
1GenTranslateV2ASR-BLEU32.3—Unverified
2GenTranslateV1ASR-BLEU30.1—Unverified
3SeamlessM4T LargeV2ASR-BLEU29.4—Unverified
4SeamlessM4T LargeASR-BLEU25.8—Unverified
5AudioPaLM2ASR-BLEU24—Unverified
6WhisperV2ASR-BLEU23.5—Unverified
7SeamlessM4T MediumASR-BLEU20.4—Unverified
#ModelMetricClaimedVerifiedStatus
1SeamlessM4T LargeASR-BLEU36.5—Unverified
2SeamlessM4T MediumASR-BLEU28.1—Unverified