| Yi: Open Foundation Models by 01.AI | Mar 7, 2024 | AttributeChatbot | CodeCode Available | 9 |
| Scaling Granite Code Models to 128K Context | Jul 18, 2024 | 2k4k | CodeCode Available | 4 |
| Retrieval Head Mechanistically Explains Long-Context Factuality | Apr 24, 2024 | Continual PretrainingHallucination | CodeCode Available | 3 |
| Data Engineering for Scaling Language Models to 128K Context | Feb 15, 2024 | 4kContinual Pretraining | CodeCode Available | 3 |
| Rho-1: Not All Tokens Are What You Need | Apr 11, 2024 | AllContinual Pretraining | CodeCode Available | 3 |
| MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning | May 20, 2024 | Continual PretrainingMathematical Reasoning | CodeCode Available | 3 |
| Towards Lifelong Learning of Large Language Models: A Survey | Jun 10, 2024 | Continual PretrainingIncremental Learning | CodeCode Available | 2 |
| A Practitioner's Guide to Continual Multimodal Pretraining | Aug 26, 2024 | Continual LearningContinual Pretraining | CodeCode Available | 2 |
| Continual Training of Language Models for Few-Shot Learning | Oct 11, 2022 | Continual LearningContinual Pretraining | CodeCode Available | 2 |
| Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts | Feb 12, 2024 | Continual PretrainingGSM8K | CodeCode Available | 2 |
| Continual Pre-training of Language Models | Feb 7, 2023 | Continual LearningContinual Pretraining | CodeCode Available | 2 |
| Effective Long-Context Scaling of Foundation Models | Sep 27, 2023 | Continual PretrainingLanguage Modeling | CodeCode Available | 2 |
| TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining | Apr 2, 2025 | Continual LearningContinual Pretraining | CodeCode Available | 1 |
| ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning | Dec 30, 2020 | Continual PretrainingLanguage Modelling | CodeCode Available | 1 |
| Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining | May 30, 2024 | Continual PretrainingContrastive Learning | CodeCode Available | 1 |
| Towards Geospatial Foundation Models via Continual Pretraining | Feb 9, 2023 | Change DetectionContinual Pretraining | CodeCode Available | 1 |
| NyayaAnumana & INLegalLlama: The Largest Indian Legal Judgment Prediction Dataset and Specialized Language Model for Enhanced Decision Analysis | Dec 11, 2024 | Continual PretrainingLanguage Modeling | CodeCode Available | 1 |
| Continual Pre-Training Mitigates Forgetting in Language and Vision | May 19, 2022 | Continual LearningContinual Pretraining | CodeCode Available | 1 |
| Demystifying Domain-adaptive Post-training for Financial LLMs | Jan 9, 2025 | Continual PretrainingDomain Adaptation | CodeCode Available | 1 |
| CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation | Aug 14, 2023 | Continual LearningContinual Pretraining | CodeCode Available | 1 |
| CTP:Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation | Jan 1, 2023 | Continual LearningContinual Pretraining | CodeCode Available | 1 |
| Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning | Sep 10, 2021 | Continual PretrainingContrastive Learning | CodeCode Available | 1 |
| On the Robustness of Reading Comprehension Models to Entity Renaming | Oct 16, 2021 | Continual PretrainingMachine Reading Comprehension | CodeCode Available | 1 |
| ChuXin: 1.6B Technical Report | May 8, 2024 | Continual PretrainingLanguage Modeling | —Unverified | 0 |
| AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy | Sep 29, 2024 | AstronomyBenchmarking | —Unverified | 0 |
