Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 15201–15250 of 17610 papers

Title	Date	Tasks	Status	Hype
Relaxed Softmax for learning from Positive and Unlabeled data	Sep 17, 2019	Density EstimationLanguage Modeling	—Unverified	0
Ludwig: a type-based declarative deep learning toolbox	Sep 17, 2019	DecoderDeep Learning	CodeCode Available	3
Character-Centric Storytelling	Sep 17, 2019	Language ModelingLanguage Modelling	—Unverified	0
Fast transcription of speech in low-resource languages	Sep 16, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Bridging Visual Perception with Contextual Semantics for Understanding Robot Manipulation Tasks	Sep 16, 2019	AttributeCommon Sense Reasoning	—Unverified	0
Global Autoregressive Models for Data-Efficient Sequence Learning	Sep 16, 2019	Language ModellingSmall Data Image Classification	CodeCode Available	0
BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle	Sep 16, 2019	Abstractive Text SummarizationExtractive Summarization	—Unverified	0
A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning	Sep 16, 2019	Code CompletionLanguage Modelling	—Unverified	0
Short-Text Classification Using Unsupervised Keyword Expansion	Sep 16, 2019	ClassificationGeneral Classification	—Unverified	0
TiM-DNN: Ternary in-Memory accelerator for Deep Neural Networks	Sep 15, 2019	GPUImage Classification	—Unverified	0
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing	Sep 15, 2019	Cross-Lingual TransferDependency Parsing	CodeCode Available	0
Representation Learning in Geology and GilBERT	Sep 14, 2019	Language ModelingLanguage Modelling	—Unverified	0
Semantic Structure Extraction for Spreadsheet Tables with a Multi-task Learning Architecture	Sep 14, 2019	Language ModelingLanguage Modelling	—Unverified	0
Ouroboros: On Accelerating Training of Transformer-Based Language Models	Sep 14, 2019	Language ModelingLanguage Modelling	CodeCode Available	1
Tree Transformer: Integrating Tree Structures into Self-Attention	Sep 14, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Speculative Beam Search for Simultaneous Translation	Sep 12, 2019	Language ModelingLanguage Modelling	—Unverified	0
Differentially Private Meta-Learning	Sep 12, 2019	Federated LearningFew-Shot Learning	—Unverified	0
CTRL: A Conditional Transformer Language Model for Controllable Generation	Sep 11, 2019	Language ModelingLanguage Modelling	CodeCode Available	1
Dynamic Fusion: Attentional Language Model for Neural Machine Translation	Sep 11, 2019	Language ModelingLanguage Modelling	—Unverified	0
BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding	Sep 11, 2019	Instance SegmentationLanguage Modeling	CodeCode Available	0
Learning Dynamic Author Representations with Temporal Language Models	Sep 11, 2019	Information RetrievalLanguage Modeling	CodeCode Available	0
An Evalutation of Programming Language Models' performance on Software Defect Detection	Sep 10, 2019	Defect DetectionLanguage Modeling	CodeCode Available	0
Multimodal Embeddings from Language Models	Sep 10, 2019	Emotion RecognitionLanguage Modeling	CodeCode Available	0
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning	Sep 10, 2019	Cross-Lingual Document ClassificationDocument Classification	CodeCode Available	1
Countering Language Drift via Visual Grounding	Sep 10, 2019	Language ModelingLanguage Modelling	—Unverified	0
Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?	Sep 9, 2019	domain classificationLanguage Modeling	—Unverified	0
Span Selection Pre-training for Question Answering	Sep 9, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Knowledge Enhanced Contextual Word Representations	Sep 9, 2019	Entity LinkingEntity Typing	CodeCode Available	0
LAMOL: LAnguage MOdeling for Lifelong Language Learning	Sep 7, 2019	Continual LearningLanguage Modeling	CodeCode Available	0
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models	Sep 7, 2019	Abstractive Text SummarizationDocument Summarization	CodeCode Available	0
KG-BERT: BERT for Knowledge Graph Completion	Sep 7, 2019	Knowledge Graph CompletionKnowledge Graphs	CodeCode Available	0
Auto-GNN: Neural Architecture Search of Graph Neural Networks	Sep 7, 2019	Language ModelingLanguage Modelling	—Unverified	0
Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression	Sep 7, 2019	Language ModellingReading Comprehension	—Unverified	0
Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes	Sep 6, 2019	General ClassificationLanguage Modeling	CodeCode Available	1
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity	Sep 5, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Robustness to Modification with Shared Words in Paraphrase Identification	Sep 5, 2019	Language ModelingLanguage Modelling	—Unverified	0
TabFact: A Large-scale Dataset for Table-based Fact Verification	Sep 5, 2019	16kFact Checking	CodeCode Available	0
Semantics-aware BERT for Language Understanding	Sep 5, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Distributionally Robust Language Modeling	Sep 4, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
PaLM: A Hybrid Parser and Language Model	Sep 4, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Mogrifier LSTM	Sep 4, 2019	Language Modelling	CodeCode Available	0
Neural Linguistic Steganography	Sep 3, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Language Models as Knowledge Bases?	Sep 3, 2019	Language ModellingOpen-Domain Question Answering	CodeCode Available	0
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives	Sep 3, 2019	Language ModelingLanguage Modelling	—Unverified	0
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks	Sep 3, 2019	Cross-Lingual Natural Language InferenceCross-Lingual Question Answering	—Unverified	0
Deep Equilibrium Models	Sep 3, 2019	Language ModelingLanguage Modelling	CodeCode Available	1
Automatic Argument Quality Assessment -- New Datasets and Methods	Sep 3, 2019	Language ModelingLanguage Modelling	—Unverified	0
Brain2Char: A Deep Architecture for Decoding Text from Brain Recordings	Sep 3, 2019	Language Modelling	—Unverified	0
The Woman Worked as a Babysitter: On Biases in Language Generation	Sep 3, 2019	Language ModelingLanguage Modelling	CodeCode Available	1
Subword Language Model for Query Auto-Completion	Sep 2, 2019	Language ModelingLanguage Modelling	CodeCode Available	0

Show:10 25 50

← PrevPage 305 of 353Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified