Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 8151–8175 of 17610 papers

Title	Date	Tasks	Status	Hype
The Impact of Depth on Compositional Generalization in Transformer Language Models	Oct 30, 2023	Language ModelingLanguage Modelling	—Unverified	0
Generative retrieval-augmented ontologic graph and multi-agent strategies for interpretive large language model-based materials design	Oct 30, 2023	Code GenerationLanguage Modeling	—Unverified	0
Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning	Oct 30, 2023	Emotion RecognitionLanguage Modelling	—Unverified	0
FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction	Oct 30, 2023	Click-Through Rate PredictionContrastive Learning	CodeCode Available	1
BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing	Oct 30, 2023	Language ModellingMulti-Task Learning	—Unverified	0
Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Models	Oct 30, 2023	Language ModellingRetrieval	—Unverified	0
BTRec: BERT-Based Trajectory Recommendation for Personalized Tours	Oct 30, 2023	Language ModellingSentence	CodeCode Available	0
MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models	Oct 30, 2023	Language ModelingLanguage Modelling	CodeCode Available	1
Integrating Pre-trained Language Model into Neural Machine Translation	Oct 30, 2023	Language ModelingLanguage Modelling	—Unverified	0
Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck	Oct 30, 2023	Language ModelingLanguage Modelling	CodeCode Available	0
Generating Medical Prescriptions with Conditional Transformer	Oct 30, 2023	2kLanguage Modelling	CodeCode Available	0
Improving Input-label Mapping with Demonstration Replay for In-context Learning	Oct 30, 2023	In-Context LearningLanguage Modeling	—Unverified	0
Large Trajectory Models are Scalable Motion Predictors and Planners	Oct 30, 2023	Autonomous DrivingLanguage Modeling	CodeCode Available	2
Skywork: A More Open Bilingual Foundation Model	Oct 30, 2023	Language ModelingLanguage Modelling	CodeCode Available	3
'Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion	Oct 30, 2023	Language Modelling	—Unverified	0
Musical Form Generation	Oct 30, 2023	FormLanguage Modeling	—Unverified	0
ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense	Oct 30, 2023	Common Sense ReasoningLanguage Modeling	CodeCode Available	0
LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses	Oct 30, 2023	FormLanguage Modeling	CodeCode Available	1
EHRTutor: Enhancing Patient Understanding of Discharge Instructions	Oct 30, 2023	Conversational Question AnsweringLanguage Modeling	—Unverified	0
Adapter Pruning using Tropical Characterization	Oct 30, 2023	Language ModelingLanguage Modelling	—Unverified	0
Counterfactually Probing Language Identity in Multilingual Models	Oct 29, 2023	counterfactualLanguage Modeling	CodeCode Available	0
A Unique Training Strategy to Enhance Language Models Capabilities for Health Mention Detection from Social Media Content	Oct 29, 2023	Contrastive LearningLanguage Modelling	—Unverified	0
Unified Representation for Non-compositional and Compositional Expressions	Oct 29, 2023	Language ModelingLanguage Modelling	CodeCode Available	0
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding	Oct 29, 2023	FormLanguage Modelling	CodeCode Available	1
MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansion	Oct 29, 2023	Information RetrievalLanguage Modelling	CodeCode Available	0

Show:10 25 50

← PrevPage 327 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified