| MPNet: Masked and Permuted Pre-training for Language Understanding | Apr 20, 2020 | Language ModelingLanguage Modelling | CodeCode Available | 2 | 5 |
| MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining | Dec 29, 2023 | GPULanguage Modeling | CodeCode Available | 2 | 5 |
| Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrieval | Oct 27, 2022 | Language ModelingLanguage Modelling | CodeCode Available | 2 | 5 |
| GPT or BERT: why not both? | Oct 31, 2024 | Causal Language ModelingLanguage Modeling | CodeCode Available | 2 | 5 |
| BMFM-RNA: An Open Framework for Building and Evaluating Transcriptomic Foundation Models | Jun 17, 2025 | BenchmarkingLanguage Modeling | CodeCode Available | 2 | 5 |
| Deep Bidirectional Language-Knowledge Graph Pretraining | Oct 17, 2022 | Common Sense ReasoningKnowledge Graphs | CodeCode Available | 2 | 5 |
| LinkBERT: Pretraining Language Models with Document Links | Mar 29, 2022 | Document ClassificationLanguage Modeling | CodeCode Available | 2 | 5 |
| RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder | May 24, 2022 | DecoderInformation Retrieval | CodeCode Available | 2 | 5 |
| A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models | Oct 16, 2021 | Image CaptioningLanguage Modeling | CodeCode Available | 1 | 5 |
| AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs | Jul 29, 2024 | Bilevel OptimizationLanguage Modelling | CodeCode Available | 1 | 5 |
