SOTAVerified

Continual Pretraining

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–50 of 70 papers

TitleStatusHype
Yi: Open Foundation Models by 01.AICode9
Scaling Granite Code Models to 128K ContextCode4
MoRA: High-Rank Updating for Parameter-Efficient Fine-TuningCode3
Retrieval Head Mechanistically Explains Long-Context FactualityCode3
Rho-1: Not All Tokens Are What You NeedCode3
Data Engineering for Scaling Language Models to 128K ContextCode3
A Practitioner's Guide to Continual Multimodal PretrainingCode2
Towards Lifelong Learning of Large Language Models: A SurveyCode2
Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical TextsCode2
Effective Long-Context Scaling of Foundation ModelsCode2
Continual Pre-training of Language ModelsCode2
Continual Training of Language Models for Few-Shot LearningCode2
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM PretrainingCode1
Demystifying Domain-adaptive Post-training for Financial LLMsCode1
NyayaAnumana & INLegalLlama: The Largest Indian Legal Judgment Prediction Dataset and Specialized Language Model for Enhanced Decision AnalysisCode1
Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation PretrainingCode1
CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology PreservationCode1
Towards Geospatial Foundation Models via Continual PretrainingCode1
CTP:Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology PreservationCode1
Continual Pre-Training Mitigates Forgetting in Language and VisionCode1
On the Robustness of Reading Comprehension Models to Entity RenamingCode1
Efficient Contrastive Learning via Novel Data Augmentation and Curriculum LearningCode1
ECONET: Effective Continual Pretraining of Language Models for Event Temporal ReasoningCode1
Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content—0
LLaVA-c: Continual Improved Visual Instruction Tuning—0
Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM EvaluationCode0
A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLPCode0
Enhance Mobile Agents Thinking Process Via Iterative Preference Learning—0
Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning—0
Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language—0
Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them—0
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling—0
AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text—0
Robust Data Watermarking in Language Models by Injecting Fictitious KnowledgeCode0
Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study—0
Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models—0
Alchemy: Amplifying Theorem-Proving Capability through Symbolic MutationCode0
The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging—0
DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining—0
AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy—0
LangSAMP: Language-Script Aware Multilingual PretrainingCode0
Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning ApproachCode0
RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining—0
Bilingual Adaptation of Monolingual Foundation Models—0
70B-parameter large language models in Japanese medical question-answering—0
Open Generative Large Language Models for Galician—0
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective—0
BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM—0
LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models—0
Cross-sensor self-supervised training and alignment for remote sensing—0
Show:102550
← PrevPage 1 of 2Next →
All datasetsACL ARCAG NewsSciERC

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1DASF1 (macro)0.69—Unverified
#ModelMetricClaimedVerifiedStatus
1CPTF1 - macro63.77—Unverified
#ModelMetricClaimedVerifiedStatus
1DASF1 (macro)0.71—Unverified