Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2301–2325 of 17610 papers

Title	Date	Tasks	Status	Hype	Score
Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little Cost	May 1, 2022	Contrastive LearningLanguage Modeling	CodeCode Available	1	5
A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysis	Apr 16, 2022	Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)	CodeCode Available	1	5
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic	Feb 20, 2024	ArabicMMLULanguage Model Evaluation	CodeCode Available	1	5
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models	Oct 16, 2021	Image CaptioningLanguage Modeling	CodeCode Available	1	5
In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning	Aug 8, 2023	In-Context LearningLanguage Modeling	CodeCode Available	1	5
Crafting Large Language Models for Enhanced Interpretability	Jul 5, 2024	Language ModelingLanguage Modelling	CodeCode Available	1	5
4-bit Shampoo for Memory-Efficient Network Training	May 28, 2024	image-classificationImage Classification	CodeCode Available	1	5
CrAM: A Compression-Aware Minimizer	Jul 28, 2022	GPUImage Classification	CodeCode Available	1	5
Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost	Mar 15, 2022	Contrastive LearningLanguage Modeling	CodeCode Available	1	5
In-context Autoencoder for Context Compression in a Large Language Model	Jul 13, 2023	GPULanguage Modeling	CodeCode Available	1	5
Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring	May 14, 2024	Language ModelingLanguage Modelling	CodeCode Available	1	5
INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model	Jul 23, 2024	Language ModelingLanguage Modelling	CodeCode Available	1	5
AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies	Aug 13, 2024	Language ModellingMixture-of-Experts	CodeCode Available	1	5
CPLLM: Clinical Prediction with Large Language Models	Sep 20, 2023	Disease PredictionLanguage Modeling	CodeCode Available	1	5
Improving Transformer Optimization Through Better Initialization	Jan 1, 2020	DecoderLanguage Modeling	CodeCode Available	1	5
Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation	Mar 18, 2021	Bilingual Lexicon InductionLanguage Modeling	CodeCode Available	1	5
CPM: A Large-scale Generative Chinese Pre-trained Language Model	Dec 1, 2020	Cloze TestLanguage Modeling	CodeCode Available	1	5
Improving Transformer Optimization Through Better Initialization	Jan 1, 2020	DecoderLanguage Modeling	CodeCode Available	1	5
CoVR-2: Automatic Data Construction for Composed Video Retrieval	Aug 28, 2023	Composed Image Retrieval (CoIR)Composed Video Retrieval (CoVR)	CodeCode Available	1	5
3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding	Jan 14, 2025	Language ModelingLanguage Modelling	CodeCode Available	1	5
Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach	Sep 16, 2024	Language ModelingLanguage Modelling	CodeCode Available	1	5
A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19	Jun 19, 2020	ChatbotLanguage Modeling	CodeCode Available	1	5
Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval	Jul 31, 2022	Knowledge DistillationLanguage Modeling	CodeCode Available	1	5
CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation	Sep 13, 2021	DecoderDenoising	CodeCode Available	1	5
Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change	Oct 31, 2022	Language ModelingLanguage Modelling	CodeCode Available	1	5

Show:10 25 50

← PrevPage 93 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified