SOTAVerified

Speech-to-Speech Translation

Speech-to-speech translation (S2ST) consists on translating speech from one language to speech in another language. This can be done with a cascade of automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis sub-systems, which is text-centric. Recently, works on S2ST without relying on intermediate text representation is emerging.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 51–100 of 117 papers

TitleStatusHype
NAIST Simultaneous Speech Translation System for IWSLT 2024—0
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling—0
Phi-Omni-ST: A multimodal language model for direct speech-to-speech translation—0
PolySinger: Singing-Voice to Singing-Voice Translation from English to Japanese—0
PolyVoice: Language Models for Speech to Speech Translation—0
Portable Speech-to-Speech Translation on an Android Smartphone: The MFLTS System—0
Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems—0
What does it take to get state of the art in simultaneous speech-to-speech translation?—0
A Case Study on Filtering for End-to-End Speech Translation—0
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation—0
Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation—0
Assessing Evaluation Metrics for Speech-to-Speech Translation—0
AudioPaLM: A Large Language Model That Can Speak and Listen—0
A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation—0
Automatic Extraction of Parallel Speech Corpora from Dubbed Movies—0
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation—0
Balancing Speech Understanding and Generation Using Continual Pre-training for Codec-based Speech LLM—0
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?—0
Connecting Voices: LoReSpeech as a Low-Resource Speech Parallel Corpus—0
Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis—0
CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning—0
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation—0
Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation—0
Direct Punjabi to English speech translation using discrete units—0
Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention—0
Direct Speech-to-Speech Neural Machine Translation: A Survey—0
Direct Speech to Speech Translation: A Review—0
Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features—0
Direct Text to Speech Translation System using Acoustic Units—0
Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing—0
Textless Speech-to-Speech Translation on Real Data—0
Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens—0
The HW-TSC’s Speech to Speech Translation System for IWSLT 2022 Evaluation—0
Towards Multilingual Conversations in the Medical Domain: Development of Multilingual Medical Data and A Network-based ASR System—0
TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data—0
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation—0
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation—0
Translatotron 3: Speech to Speech Translation with Monolingual Data—0
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units—0
UWSpeech: Speech to Speech Translation for Unwritten Languages—0
Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs—0
Prosodic Alignment for off-screen automatic dubbing—0
Real-time Incremental Speech-to-Speech Translation of Dialogs—0
S2ST-Omni: An Efficient and Scalable Multilingual Speech-to-Speech Translation Framework via Seamless Speech-Text Alignment and Streaming Speech Generation—0
SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation—0
SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought—0
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation—0
Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS—0
SimulTron: On-Device Simultaneous Speech to Speech Translation—0
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations—0
Show:102550
← PrevPage 2 of 3Next →
All datasetsTATFLEURS X-engCVSS

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Hokkien→En (Two-pass decoding)ASR-BLEU (Dev)13.6—Unverified
2Hokkien→En (Two-stage)ASR-BLEU (Dev)12.5—Unverified
3Hokkien→En (Three-stage)ASR-BLEU (Dev)12.5—Unverified
4Hokkien→En (Single-pass decoding)ASR-BLEU (Dev)8.8—Unverified
5En→Hokkien (Two-pass decoding)ASR-BLEU (Dev)7.8—Unverified
6En→Hokkien (Three-stage)ASR-BLEU (Dev)7.5—Unverified
7En→Hokkien (Two-stage)ASR-BLEU (Dev)7.1—Unverified
8En→Hokkien (Single-pass decoding)ASR-BLEU (Dev)6.6—Unverified
#ModelMetricClaimedVerifiedStatus
1GenTranslateV2ASR-BLEU32.3—Unverified
2GenTranslateV1ASR-BLEU30.1—Unverified
3SeamlessM4T LargeV2ASR-BLEU29.4—Unverified
4SeamlessM4T LargeASR-BLEU25.8—Unverified
5AudioPaLM2ASR-BLEU24—Unverified
6WhisperV2ASR-BLEU23.5—Unverified
7SeamlessM4T MediumASR-BLEU20.4—Unverified
#ModelMetricClaimedVerifiedStatus
1SeamlessM4T LargeASR-BLEU36.5—Unverified
2SeamlessM4T MediumASR-BLEU28.1—Unverified