Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 11476–11500 of 17610 papers

Title	Date	Tasks	Status	Hype
Language Modelling with Pixels	Jul 14, 2022	Language ModellingNamed Entity Recognition	CodeCode Available	2
Layout-Aware Information Extraction for Document-Grounded Dialogue: Dataset, Method and Demonstration	Jul 14, 2022	Language ModelingLanguage Modelling	—Unverified	0
Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages	Jul 14, 2022	Language ModelingLanguage Modelling	CodeCode Available	0
Recurrent Memory Transformer	Jul 14, 2022	Language ModelingLanguage Modelling	CodeCode Available	2
Language models show human-like content effects on reasoning tasks	Jul 14, 2022	Language ModellingLogical Reasoning	CodeCode Available	0
Scene Text Recognition with Permuted Autoregressive Sequence Models	Jul 14, 2022	Language ModelingLanguage Modelling	CodeCode Available	2
Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model	Jul 14, 2022	Data AugmentationData-to-Text Generation	—Unverified	0
TRIE++: Towards End-to-End Information Extraction from Visually Rich Documents	Jul 14, 2022	global-optimizationLanguage Modelling	—Unverified	0
A Transfer Learning Based Model for Text Readability Assessment in German	Jul 13, 2022	Language ModelingLanguage Modelling	—Unverified	0
N-Grammer: Augmenting Transformers with latent n-grams	Jul 13, 2022	Common Sense ReasoningCoreference Resolution	CodeCode Available	4
Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS	Jul 13, 2022	Language ModelingLanguage Modelling	—Unverified	0
A Novel DeBERTa-based Model for Financial Question Answering Task	Jul 12, 2022	Language ModelingLanguage Modelling	—Unverified	0
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action	Jul 10, 2022	Instruction FollowingLanguage Modeling	CodeCode Available	2
Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognition	Jul 9, 2022	Language ModelingLanguage Modelling	—Unverified	0
The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applications	Jul 8, 2022	Binary ClassificationLanguage Modeling	CodeCode Available	1
Hidden Schema Networks	Jul 8, 2022	DecoderLanguage Modelling	—Unverified	0
Predicting Opinion Dynamics via Sociologically-Informed Neural Networks	Jul 7, 2022	Language ModellingSociology	CodeCode Available	1
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation	Jul 7, 2022	Abstractive Text SummarizationLanguage Modeling	CodeCode Available	0
Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps	Jul 7, 2022	Language Modelling	—Unverified	0
A Large Scale Search Dataset for Unbiased Learning to Rank	Jul 7, 2022	Causal DiscoveryLanguage Modelling	CodeCode Available	1
Aspect-Based Sentiment Analysis using Local Context Focus Mechanism with DeBERTa	Jul 6, 2022	Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)	—Unverified	0
Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning	Jul 6, 2022	Language ModelingLanguage Modelling	CodeCode Available	0
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval	Jul 6, 2022	Language ModelingLanguage Modelling	—Unverified	0
Text Enriched Sparse Hyperbolic Graph Convolutional Networks	Jul 6, 2022	Language ModellingLink Prediction	—Unverified	0
MIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering	Jul 5, 2022	Cross-Lingual Question AnsweringDecoder	—Unverified	0

Show:10 25 50

← PrevPage 460 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified