Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 4701–4725 of 17610 papers

Title	Date	Tasks	Status	Hype
MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting	Jun 25, 2024	Language ModelingLanguage Modelling	—Unverified	0
Discrete Diffusion Language Model for Long Text Summarization	Jun 25, 2024	Abstractive Text SummarizationDecoder	—Unverified	0
High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model	Jun 25, 2024	Computational EfficiencyLanguage Modeling	—Unverified	0
LABOR-LLM: Language-Based Occupational Representations with Large Language Models	Jun 25, 2024	In-Context LearningJob Prediction	—Unverified	0
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment	Jun 25, 2024	DiversityLanguage Modeling	CodeCode Available	0
CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design	Jun 25, 2024	Language ModelingLanguage Modelling	CodeCode Available	0
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference	Jun 25, 2024	Language ModelingLanguage Modelling	CodeCode Available	1
Multi-property Steering of Large Language Models with Dynamic Activation Composition	Jun 25, 2024	Language ModelingLanguage Modelling	CodeCode Available	1
Accelerating Clinical Evidence Synthesis with Large Language Models	Jun 25, 2024	Language Modelling	—Unverified	0
Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training	Jun 25, 2024	DenoisingLanguage Modeling	CodeCode Available	0
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale	Jun 25, 2024	ARCLanguage Modeling	CodeCode Available	1
CogMG: Collaborative Augmentation Between Large Language Model and Knowledge Graph	Jun 25, 2024	Knowledge Graph CompletionKnowledge Graphs	CodeCode Available	1
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients	Jun 25, 2024	GPULanguage Modeling	CodeCode Available	1
Native Design Bias: Studying the Impact of English Nativeness on Language Model Performance	Jun 25, 2024	DiversityLanguage Modeling	CodeCode Available	0
Understanding Language Model Circuits through Knowledge Editing	Jun 25, 2024	knowledge editingLanguage Modeling	—Unverified	0
Semi-supervised classification of dental conditions in panoramic radiographs using large language model and instance segmentation: A real-world dataset evaluation	Jun 25, 2024	DiagnosticInstance Segmentation	—Unverified	0
VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation	Jun 25, 2024	ARCBenchmarking	CodeCode Available	0
TRAWL: Tensor Reduced and Approximated Weights for Large Language Models	Jun 25, 2024	Language ModellingLarge Language Model	CodeCode Available	0
Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classification	Jun 25, 2024	Contrastive Learningfew-shot-htc	CodeCode Available	1
Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization?	Jun 25, 2024	Decision MakingLanguage Modeling	CodeCode Available	0
AG-LSEC: Audio Grounded Lexical Speaker Error Correction	Jun 25, 2024	Language ModelingLanguage Modelling	—Unverified	0
Enhancing Tool Retrieval with Iterative Feedback from Large Language Models	Jun 25, 2024	In-Context LearningLanguage Modeling	CodeCode Available	0
A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR	Jun 25, 2024	Language ModelingLanguage Modelling	—Unverified	0
Human-Object Interaction from Human-Level Instructions	Jun 25, 2024	Common Sense ReasoningHuman-Object Interaction Detection	—Unverified	0
Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks	Jun 25, 2024	Language ModelingLanguage Modelling	—Unverified	0

Show:10 25 50

← PrevPage 189 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified