Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 15151–15200 of 17610 papers

Title	Date	Tasks	Status	Hype
The Detection of Distributional Discrepancy for Text Generation	Sep 28, 2019	Language ModelingLanguage Modelling	—Unverified	0
End-to-End Code-Switching ASR for Low-Resourced Language Pairs	Sep 27, 2019	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Improving Pre-Trained Multilingual Models with Vocabulary Expansion	Sep 26, 2019	Language ModelingLanguage Modelling	—Unverified	0
Biomedical relation extraction with pre-trained language representations and minimal task-specific architecture	Sep 26, 2019	Language ModelingLanguage Modelling	—Unverified	0
DARTS: Dialectal Arabic Transcription System	Sep 26, 2019	Language ModelingLanguage Modelling	—Unverified	0
Anchor & Transform: Learning Sparse Representations of Discrete Objects	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
Forecasting Deep Learning Dynamics with Applications to Hyperparameter Tuning	Sep 25, 2019	Deep LearningLanguage Modeling	—Unverified	0
EINS: Long Short-Term Memory with Extrapolated Input Network Simplification	Sep 25, 2019	Image GenerationImputation	—Unverified	0
Enhancing Attention with Explicit Phrasal Alignments	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
UNITER: Learning UNiversal Image-TExt Representations	Sep 25, 2019	Image-text matchingImage-text Retrieval	—Unverified	0
XD: Cross-lingual Knowledge Distillation for Polyglot Sentence Embeddings	Sep 25, 2019	Knowledge DistillationLanguage Modeling	—Unverified	0
Layer Flexible Adaptive Computation Time for Recurrent Neural Networks	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
Self-Supervised Speech Recognition via Local Prior Matching	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
Lossless Data Compression with Transformer	Sep 25, 2019	Data CompressionLanguage Modeling	—Unverified	0
Structural Language Models for Any-Code Generation	Sep 25, 2019	C++ codeCode Generation	—Unverified	0
Recurrent Hierarchical Topic-Guided Neural Language Models	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
Putting Machine Translation in Context with the Noisy Channel Model	Sep 25, 2019	Document TranslationLanguage Modeling	—Unverified	0
SoftAdam: Unifying SGD and Adam for better stochastic gradient descent	Sep 25, 2019	image-classificationImage Classification	—Unverified	0
Sparse Transformer: Concentrated Attention Through Explicit Selection	Sep 25, 2019	Image CaptioningLanguage Modeling	—Unverified	0
Scalable Neural Learning for Verifiable Consistency with Temporal Specifications	Sep 25, 2019	Adversarial RobustnessLanguage Modeling	—Unverified	0
Interpretable Network Structure for Modeling Contextual Dependency	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
ASGen: Answer-containing Sentence Generation to Pre-Train Question Generator for Scale-up Data in Question Answering	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
Group-Transformer: Towards A Lightweight Character-level Language Model	Sep 25, 2019	Language ModelingLanguage Modelling	—Unverified	0
Extremely Small BERT Models from Mixed-Vocabulary Training	Sep 25, 2019	Knowledge DistillationLanguage Modelling	—Unverified	0
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models	Sep 25, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Reducing Transformer Depth on Demand with Structured Dropout	Sep 25, 2019	Language ModelingLanguage Modelling	CodeCode Available	1
UNITER: UNiversal Image-TExt Representation Learning	Sep 25, 2019	Image-text matchingImage-text Retrieval	CodeCode Available	1
Understanding Semantics from Speech Through Pre-training	Sep 24, 2019	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Code-switching Language Modeling With Bilingual Word Embeddings: A Case Study for Egyptian Arabic-English	Sep 24, 2019	Language ModelingLanguage Modelling	—Unverified	0
TinyBERT: Distilling BERT for Natural Language Understanding	Sep 23, 2019	Knowledge DistillationLanguage Modelling	CodeCode Available	0
Adapting Language Models for Non-Parallel Author-Stylized Rewriting	Sep 22, 2019	DecoderDenoising	—Unverified	0
Inducing Constituency Trees through Neural Machine Translation	Sep 22, 2019	Language ModelingLanguage Modelling	—Unverified	0
Understanding and Robustifying Differentiable Architecture Search	Sep 20, 2019	Disparity Estimationimage-classification	CodeCode Available	0
Creative GANs for generating poems, lyrics, and metaphors	Sep 20, 2019	Generative Adversarial NetworkLanguage Modeling	CodeCode Available	0
A Critical Analysis of Biased Parsers in Unsupervised Parsing	Sep 20, 2019	Language ModelingLanguage Modelling	CodeCode Available	1
Understanding Architectures Learnt by Cell-based Neural Architecture Search	Sep 20, 2019	image-classificationImage Classification	CodeCode Available	0
A Comparison of Hybrid and End-to-End Models for Syllable Recognition	Sep 19, 2019	Language ModelingLanguage Modelling	—Unverified	0
How Additional Knowledge can Improve Natural Language Commonsense Question Answering?	Sep 19, 2019	ArticlesLanguage Modeling	—Unverified	0
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models	Sep 19, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
A Random Gossip BMUF Process for Neural Language Modeling	Sep 19, 2019	GPULanguage Modeling	—Unverified	0
Self-Training for End-to-End Speech Recognition	Sep 19, 2019	DiversityLanguage Modeling	—Unverified	0
Pre-trained Language Model for Biomedical Question Answering	Sep 18, 2019	Language ModelingLanguage Modelling	CodeCode Available	0
Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences	Sep 18, 2019	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Espresso: A Fast End-to-end Neural Speech Recognition Toolkit	Sep 18, 2019	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	CodeCode Available	1
Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes	Sep 18, 2019	Language Modelling	CodeCode Available	0
Enriching BERT with Knowledge Graph Embeddings for Document Classification	Sep 18, 2019	ClassificationDescriptive	CodeCode Available	0
Fine-Tuning Language Models from Human Preferences	Sep 18, 2019	DescriptiveLanguage Modelling	CodeCode Available	3
SUPP.AI: Finding Evidence for Supplement-Drug Interactions	Sep 17, 2019	ArticlesGeneral Classification	CodeCode Available	0
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism	Sep 17, 2019	GPULAMBADA	CodeCode Available	2
Pointer-based Fusion of Bilingual Lexicons into Neural Machine Translation	Sep 17, 2019	DecoderLanguage Modeling	CodeCode Available	0

Show:10 25 50

← PrevPage 304 of 353Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified