Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3401–3425 of 17610 papers

Title	Date	Tasks	Status	Hype
Efficient Nearest Neighbor Language Models	Sep 9, 2021	Domain AdaptationLanguage Modeling	CodeCode Available	1
Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes	Sep 1, 2023	Language ModelingLanguage Modelling	CodeCode Available	1
Efficient Neural Architecture Search via Parameter Sharing	Feb 9, 2018	GPULanguage Modelling	CodeCode Available	1
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization	Oct 5, 2023	AllLanguage Modeling	CodeCode Available	1
Adversarial Training for Aspect-Based Sentiment Analysis with BERT	Jan 30, 2020	Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)	CodeCode Available	1
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering	Apr 13, 2021	Common Sense ReasoningGraph Representation Learning	CodeCode Available	1
Efficiently Modeling Long Sequences with Structured State Spaces	Oct 31, 2021	Data AugmentationLanguage Modeling	CodeCode Available	1
Efficient OCR for Building a Diverse Digital History	Apr 5, 2023	DiversityImage Retrieval	CodeCode Available	1
Efficient Hierarchical Domain Adaptation for Pretrained Language Models	Dec 16, 2021	Domain AdaptationLanguage Modeling	CodeCode Available	1
Efficient Long Sequence Modeling via State Space Augmented Transformer	Dec 15, 2022	Computational EfficiencyDecoder	CodeCode Available	1
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting	Oct 17, 2023	Language ModellingSensitivity	CodeCode Available	1
Connecting Language and Vision for Natural Language-Based Vehicle Retrieval	May 31, 2021	Language ModellingManagement	CodeCode Available	1
Efficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generation	Apr 4, 2025	ClusteringHallucination	CodeCode Available	1
Efficient Content-Based Sparse Attention with Routing Transformers	Mar 12, 2020	Image GenerationLanguage Modeling	CodeCode Available	1
Efficient Online Data Mixing For Language Model Pre-Training	Dec 5, 2023	Language ModelingLanguage Modelling	CodeCode Available	1
ELECTRAMed: a new pre-trained language representation model for biomedical NLP	Apr 19, 2021	Drug–drug Interaction ExtractionLanguage Modeling	CodeCode Available	1
Effective Use of Graph Convolution Network and Contextual Sub-Tree forCommodity News Event Extraction	Sep 27, 2021	Event ExtractionLanguage Modeling	CodeCode Available	1
Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering	May 2, 2020	Knowledge GraphsLanguage Modeling	CodeCode Available	1
Effective Sequence-to-Sequence Dialogue State Tracking	Aug 31, 2021	Dialogue State TrackingLanguage Modeling	CodeCode Available	1
Effective Use of Graph Convolution Network and Contextual Sub-Tree for Commodity News Event Extraction	Nov 1, 2021	Event ExtractionLanguage Modeling	CodeCode Available	1
Effectiveness of self-supervised pre-training for speech recognition	Nov 10, 2019	Language ModellingQuantization	CodeCode Available	1
Log Parsing: How Far Can ChatGPT Go?	Jun 2, 2023	Language ModelingLanguage Modelling	CodeCode Available	1
Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts	Dec 12, 2022	Language ModellingWord Embeddings	CodeCode Available	1
Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images	May 31, 2021	Few-Shot LearningImage Classification	CodeCode Available	1
Effective Batching for Recurrent Neural Network Grammars	May 31, 2021	GPULanguage Modeling	CodeCode Available	1

Show:10 25 50

← PrevPage 137 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified