SOTAVerified

Speech-to-Speech Translation

Speech-to-speech translation (S2ST) consists on translating speech from one language to speech in another language. This can be done with a cascade of automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis sub-systems, which is text-centric. Recently, works on S2ST without relying on intermediate text representation is emerging.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 51–75 of 117 papers

TitleStatusHype
NAIST Simultaneous Speech Translation System for IWSLT 2024—0
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling—0
Phi-Omni-ST: A multimodal language model for direct speech-to-speech translation—0
PolySinger: Singing-Voice to Singing-Voice Translation from English to Japanese—0
PolyVoice: Language Models for Speech to Speech Translation—0
Portable Speech-to-Speech Translation on an Android Smartphone: The MFLTS System—0
Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems—0
What does it take to get state of the art in simultaneous speech-to-speech translation?—0
A Case Study on Filtering for End-to-End Speech Translation—0
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation—0
Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation—0
Assessing Evaluation Metrics for Speech-to-Speech Translation—0
AudioPaLM: A Large Language Model That Can Speak and Listen—0
A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation—0
Automatic Extraction of Parallel Speech Corpora from Dubbed Movies—0
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation—0
Balancing Speech Understanding and Generation Using Continual Pre-training for Codec-based Speech LLM—0
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?—0
Connecting Voices: LoReSpeech as a Low-Resource Speech Parallel Corpus—0
Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis—0
CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning—0
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation—0
Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation—0
Direct Punjabi to English speech translation using discrete units—0
Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention—0
Show:102550
← PrevPage 3 of 5Next →
All datasetsTATFLEURS X-engCVSS

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Hokkien→En (Two-pass decoding)ASR-BLEU (Dev)13.6—Unverified
2Hokkien→En (Two-stage)ASR-BLEU (Dev)12.5—Unverified
3Hokkien→En (Three-stage)ASR-BLEU (Dev)12.5—Unverified
4Hokkien→En (Single-pass decoding)ASR-BLEU (Dev)8.8—Unverified
5En→Hokkien (Two-pass decoding)ASR-BLEU (Dev)7.8—Unverified
6En→Hokkien (Three-stage)ASR-BLEU (Dev)7.5—Unverified
7En→Hokkien (Two-stage)ASR-BLEU (Dev)7.1—Unverified
8En→Hokkien (Single-pass decoding)ASR-BLEU (Dev)6.6—Unverified
#ModelMetricClaimedVerifiedStatus
1GenTranslateV2ASR-BLEU32.3—Unverified
2GenTranslateV1ASR-BLEU30.1—Unverified
3SeamlessM4T LargeV2ASR-BLEU29.4—Unverified
4SeamlessM4T LargeASR-BLEU25.8—Unverified
5AudioPaLM2ASR-BLEU24—Unverified
6WhisperV2ASR-BLEU23.5—Unverified
7SeamlessM4T MediumASR-BLEU20.4—Unverified
#ModelMetricClaimedVerifiedStatus
1SeamlessM4T LargeASR-BLEU36.5—Unverified
2SeamlessM4T MediumASR-BLEU28.1—Unverified