SOTAVerified

fairseq S2T: Fast Speech-to-Text Modeling with fairseq

2020-10-11Asian Chapter of the Association for Computational LinguisticsCode Available0· sign in to hype

Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino

arXivPDF

Code Available — Be the first to reproduce this paper.

Reproduce

Code

Abstract

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.

Tasks

Machine TranslationMulti-Task Learningspeech-recognitionSpeech RecognitionSpeech-to-TextSpeech-to-Text TranslationTransfer LearningTranslation

Benchmark Results

DatasetModelMetricClaimedVerifiedStatus
MuST-C EN->DETransformer + ASR PretrainCase-sensitive sacreBLEU22.7—Unverified
MuST-C EN->DETransformer + ASR PretrainCase-sensitive sacreBLEU22.8—Unverified

Reproductions