Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 11026–11050 of 17610 papers

Title	Date	Tasks	Status	Hype
Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios	Oct 25, 2022	Language ModelingLanguage Modelling	—Unverified	0
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models	Oct 25, 2022	Language ModelingLanguage Modelling	—Unverified	0
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence	Oct 25, 2022	Language ModelingLanguage Modelling	—Unverified	0
Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing	Oct 25, 2022	Language ModelingLanguage Modelling	CodeCode Available	1
Contrastive Search Is What You Need For Neural Text Generation	Oct 25, 2022	Contrastive LearningLanguage Modeling	CodeCode Available	2
A single-cell gene expression language model	Oct 25, 2022	DiversityLanguage Modeling	CodeCode Available	1
Dual Mechanism Priming Effects in Hindi Word Order	Oct 25, 2022	Language ModelingLanguage Modelling	—Unverified	0
Differentially Private Language Models for Secure Data Sharing	Oct 25, 2022	Language Modelling	—Unverified	0
Characterizing Verbatim Short-Term Memory in Neural Language Models	Oct 24, 2022	Language ModellingRetrieval	CodeCode Available	0
Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models	Oct 24, 2022	Language ModelingLanguage Modelling	—Unverified	0
An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks	Oct 24, 2022	Language ModelingLanguage Modelling	CodeCode Available	0
ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation	Oct 24, 2022	Language ModelingLanguage Modelling	CodeCode Available	1
Towards Unifying Reference Expression Generation and Comprehension	Oct 24, 2022	Language ModelingLanguage Modelling	CodeCode Available	0
A BERT-based Deep Learning Approach for Reputation Analysis in Social Media	Oct 23, 2022	Language ModelingLanguage Modelling	—Unverified	0
Discriminative Language Model as Semantic Consistency Scorer for Prompt-based Few-Shot Text Classification	Oct 23, 2022	Few-Shot Text ClassificationLanguage Modeling	—Unverified	0
Do Language Models Understand Measurements?	Oct 23, 2022	Language ModelingLanguage Modelling	—Unverified	0
Code4Struct: Code Generation for Few-Shot Event Structure Prediction	Oct 23, 2022	Code GenerationEvent Argument Extraction	CodeCode Available	1
Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition	Oct 23, 2022	Language Modellingnamed-entity-recognition	CodeCode Available	0
Language Model Pre-Training with Sparse Latent Typing	Oct 23, 2022	Few-shot NERLanguage Modeling	CodeCode Available	1
LMPriors: Pre-Trained Language Models as Task-Specific Priors	Oct 22, 2022	Causal InferenceCommon Sense Reasoning	—Unverified	0
NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation	Oct 22, 2022	counterfactualData Augmentation	CodeCode Available	0
Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders	Oct 22, 2022	Language ModellingRecommendation Systems	CodeCode Available	1
P^3LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training	Oct 22, 2022	Conversational Question AnsweringDecoder	—Unverified	0
PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding	Oct 22, 2022	Language ModelingLanguage Modelling	—Unverified	0
Understanding Domain Learning in Language Models Through Subpopulation Analysis	Oct 22, 2022	Language ModelingLanguage Modelling	CodeCode Available	0

Show:10 25 50

← PrevPage 442 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified