| Predicting Attention Sparsity in Transformers | Sep 24, 2021 | DecoderLanguage Modeling | —Unverified | 0 |
| MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection | Sep 16, 2021 | AttributeLanguage Modeling | CodeCode Available | 0 |
| SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations | Sep 15, 2021 | CoLAContrastive Learning | CodeCode Available | 1 |
| CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation | Sep 13, 2021 | DecoderDenoising | CodeCode Available | 1 |
| Data Efficient Masked Language Modeling for Vision and Language | Sep 5, 2021 | Language ModelingLanguage Modelling | CodeCode Available | 1 |
| Frustratingly Simple Pretraining Alternatives to Masked Language Modeling | Sep 4, 2021 | Language ModelingLanguage Modelling | CodeCode Available | 1 |
| Split-and-Rephrase in a Cross-Lingual Manner: A Complete Pipeline | Sep 1, 2021 | Language ModelingLanguage Modelling | —Unverified | 0 |
| Domain-Specific Japanese ELECTRA Model Using a Small Corpus | Sep 1, 2021 | ArticlesComputational Efficiency | —Unverified | 0 |
| CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations | Sep 1, 2021 | Emotion ClassificationLanguage Modeling | CodeCode Available | 1 |
| Sentence Bottleneck Autoencoders from Transformer Language Models | Aug 31, 2021 | DecoderDenoising | CodeCode Available | 1 |
| MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER | Aug 31, 2021 | Cross-Lingual NERData Augmentation | CodeCode Available | 1 |
| Prompt-Learning for Fine-Grained Entity Typing | Aug 24, 2021 | Entity TypingKnowledge Probing | —Unverified | 0 |
| Knowledge Perceived Multi-modal Pretraining in E-commerce | Aug 20, 2021 | Language ModelingLanguage Modelling | CodeCode Available | 1 |
| W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training | Aug 7, 2021 | Contrastive LearningLanguage Modeling | CodeCode Available | 3 |
| Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification | Aug 4, 2021 | ClassificationFew-Shot Text Classification | CodeCode Available | 1 |
| Noobs at Semeval-2021 Task 4: Masked Language Modeling for abstract answer prediction | Aug 1, 2021 | Language ModelingLanguage Modelling | —Unverified | 0 |
| Fine-Grained Emotion Prediction by Modeling Emotion Definitions | Jul 26, 2021 | Language ModelingLanguage Modelling | CodeCode Available | 0 |
| Learning to Sample Replacements for ELECTRA Pre-Training | Jun 25, 2021 | Language ModelingLanguage Modelling | —Unverified | 0 |
| Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model | Jun 24, 2021 | DecoderLanguage Modeling | —Unverified | 0 |
| SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs | Jun 18, 2021 | DecoderKnowledge Graphs | CodeCode Available | 1 |
| SAS: Self-Augmentation Strategy for Language Model Pre-training | Jun 14, 2021 | Data AugmentationLanguage Modeling | CodeCode Available | 0 |
| Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment | Jun 11, 2021 | DenoisingLanguage Modeling | CodeCode Available | 1 |
| Exploring Unsupervised Pretraining Objectives for Machine Translation | Jun 10, 2021 | DecoderLanguage Modeling | CodeCode Available | 0 |
| MST: Masked Self-Supervised Transformer for Visual Representation | Jun 10, 2021 | Language ModelingLanguage Modelling | —Unverified | 0 |
| BERTnesia: Investigating the capture and forgetting of knowledge in BERT | Jun 5, 2021 | Knowledge Base CompletionLanguage Modeling | CodeCode Available | 0 |