Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 8451–8475 of 17610 papers

Title	Date	Tasks	Status	Hype
Multimodal Large Language Model for Visual Navigation	Oct 12, 2023	Language ModelingLanguage Modelling	—Unverified	0
Mapping Memes to Words for Multimodal Hateful Meme Classification	Oct 12, 2023	Hateful Meme ClassificationLanguage Modeling	CodeCode Available	1
QASiNa: Religious Domain Question Answering using Sirah Nabawiyah	Oct 12, 2023	Language ModellingLarge Language Model	CodeCode Available	0
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models	Oct 12, 2023	Language ModellingLarge Language Model	CodeCode Available	7
Toward Joint Language Modeling for Speech Units and Text	Oct 12, 2023	Language ModelingLanguage Modelling	—Unverified	0
Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniques	Oct 12, 2023	In-Context LearningLanguage Modelling	—Unverified	0
Towards Evaluating Generalist Agents: An Automated Benchmark in Open World	Oct 12, 2023	BenchmarkingDiversity	CodeCode Available	1
Large Language Models for Scientific Synthesis, Inference and Explanation	Oct 12, 2023	Code GenerationLanguage Modeling	CodeCode Available	1
Language Models are Universal Embedders	Oct 12, 2023	Code SearchLanguage Modeling	CodeCode Available	1
GameGPT: Multi-agent Collaborative Framework for Game Development	Oct 12, 2023	Code GenerationHallucination	—Unverified	0
Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability	Oct 12, 2023	Causal Language ModelingIn-Context Learning	CodeCode Available	0
Expanding the Vocabulary of BERT for Knowledge Base Construction	Oct 12, 2023	Knowledge Base ConstructionKnowledge Base Population	CodeCode Available	0
Context Compression for Auto-regressive Transformers with Sentinel Tokens	Oct 12, 2023	Language ModelingLanguage Modelling	CodeCode Available	1
Harnessing Large Language Models' Empathetic Response Generation Capabilities for Online Mental Health Counselling Support	Oct 12, 2023	Empathetic Response GenerationLanguage Modeling	—Unverified	0
GraphextQA: A Benchmark for Evaluating Graph-Enhanced Large Language Models	Oct 12, 2023	Answer GenerationHallucination	CodeCode Available	0
Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning	Oct 12, 2023	Image CaptioningImage-text Retrieval	—Unverified	0
Towards Robust Multi-Modal Reasoning via Model Selection	Oct 12, 2023	Language ModellingLarge Language Model	CodeCode Available	1
DistillSpec: Improving Speculative Decoding via Knowledge Distillation	Oct 12, 2023	Knowledge DistillationLanguage Modelling	—Unverified	0
HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science	Oct 12, 2023	Language ModelingLanguage Modelling	CodeCode Available	1
On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models	Oct 11, 2023	Language ModelingLanguage Modelling	CodeCode Available	0
Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models	Oct 11, 2023	Language ModelingLanguage Modelling	—Unverified	0
Language Models As Semantic Indexers	Oct 11, 2023	Contrastive LearningInformation Retrieval	CodeCode Available	1
Measuring Feature Sparsity in Language Models	Oct 11, 2023	Language ModelingLanguage Modelling	—Unverified	0
LangNav: Language as a Perceptual Representation for Navigation	Oct 11, 2023	Image CaptioningLanguage Modeling	—Unverified	0
From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models	Oct 11, 2023	In-Context LearningInstruction Following	CodeCode Available	0

Show:10 25 50

← PrevPage 339 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified