Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 16901–16950 of 17610 papers

Title	Date	Tasks	Status	Hype
Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model	Sep 1, 2015	Language ModelingLanguage Modelling	—Unverified	0
Key Concept Identification for Medical Information Retrieval	Sep 1, 2015	Information RetrievalLanguage Modelling	—Unverified	0
Script Induction as Language Modeling	Sep 1, 2015	Language ModelingLanguage Modelling	—Unverified	0
The Forest Convolutional Network: Compositional Distributional Semantics with a Neural Chart and without Binarization	Sep 1, 2015	BinarizationLanguage Modelling	—Unverified	0
Investigating Continuous Space Language Models for Machine Translation Quality Estimation	Sep 1, 2015	Language ModellingMachine Translation	—Unverified	0
Open-Domain Name Error Detection using a Multi-Task RNN	Sep 1, 2015	Language ModellingNamed Entity Recognition (NER)	—Unverified	0
Pre-Computable Multi-Layer Neural Network Language Models	Sep 1, 2015	Language ModellingMachine Translation	—Unverified	0
Touch-Based Pre-Post-Editing of Machine Translation Output	Sep 1, 2015	Language ModellingMachine Translation	—Unverified	0
Learning Word Meanings and Grammar for Describing Everyday Activities in Smart Environments	Sep 1, 2015	Language Modelling	—Unverified	0
Reverse-engineering Language: A Study on the Semantic Compositionality of German Compounds	Sep 1, 2015	Language Modelling	—Unverified	0
Parsing English into Abstract Meaning Representation Using Syntax-Based Machine Translation	Sep 1, 2015	Abstract Meaning RepresentationAMR Parsing	—Unverified	0
Transducer Disambiguation with Sparse Topological Features	Sep 1, 2015	Language Modelling	—Unverified	0
A Neural Algorithm of Artistic Style	Aug 26, 2015	Language ModellingStyle Transfer	CodeCode Available	1
Character-Aware Neural Language Models	Aug 26, 2015	Language ModelingLanguage Modelling	CodeCode Available	1
Auto-Sizing Neural Networks: With Applications to n-gram Language Models	Aug 20, 2015	DecoderLanguage Modeling	—Unverified	0
Probabilistic Modelling of Morphologically Rich Languages	Aug 18, 2015	Language ModelingLanguage Modelling	—Unverified	0
End-to-End Attention-based Large Vocabulary Speech Recognition	Aug 18, 2015	Acoustic ModellingLanguage Modeling	CodeCode Available	0
Depth-Gated LSTM	Aug 16, 2015	Language ModelingLanguage Modelling	—Unverified	0
Online Representation Learning in Recurrent Neural Language Models	Aug 16, 2015	Language ModellingRepresentation Learning	—Unverified	0
Deep Graph Kernels	Aug 10, 2015	Graph ClassificationLanguage Modeling	—Unverified	0
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation	Aug 9, 2015	FormLanguage Modeling	CodeCode Available	0
Sublinear Partition Estimation	Aug 7, 2015	Language ModelingLanguage Modelling	CodeCode Available	0
An End-to-End Neural Network for Polyphonic Piano Music Transcription	Aug 7, 2015	Language ModelingLanguage Modelling	CodeCode Available	0
Listen, Attend and Spell	Aug 5, 2015	DecoderLanguage Modeling	CodeCode Available	1
Handwritten Text Recognition Results on the Bentham Collection with Improved Classical N-Gram-HMM methods	Aug 1, 2015	Handwritten Text RecognitionLanguage Modeling	—Unverified	0
Classifying Syntactic Categories in the Chinese Dependency Network	Aug 1, 2015	Language Modelling	—Unverified	0
Discriminative Segmental Cascades for Feature-Rich Phone Recognition	Jul 22, 2015	Language ModelingLanguage Modelling	—Unverified	0
Grid Long Short-Term Memory	Jul 6, 2015	Language ModellingMemorization	CodeCode Available	0
Dependency Recurrent Neural Language Models for Sentence Completion	Jul 5, 2015	Language ModelingLanguage Modelling	CodeCode Available	0
Evaluating distributed word representations for capturing semantics of biomedical concepts	Jul 1, 2015	ArticlesChunking	—Unverified	0
GWU-HASP-2015@QALB-2015 Shared Task: Priming Spelling Candidates with Probability	Jul 1, 2015	Language Modelling	—Unverified	0
DCU-ADAPT: Learning Edit Operations for Microblog Normalisation with the Generalised Perceptron	Jul 1, 2015	Language Modelling	—Unverified	0
Chinese Grammatical Error Diagnosis System Based on Hybrid Model	Jul 1, 2015	Grammatical Error CorrectionLanguage Acquisition	—Unverified	0
Arib@QALB-2015 Shared Task: A Hybrid Cascade Model for Arabic Spelling Error Detection and Correction	Jul 1, 2015	Language Modelling	—Unverified	0
Chinese Spelling Check System Based on N-gram Model	Jul 1, 2015	Chinese Word SegmentationLanguage Modelling	—Unverified	0
Using word embedding for bio-event extraction	Jul 1, 2015	Event ExtractionLanguage Modelling	—Unverified	0
Word Vector/Conditional Random Field-based Chinese Spelling Error Detection for SIGHAN-2015 Evaluation	Jul 1, 2015	Language Modelling	—Unverified	0
Passive and Pervasive Use of Bilingual Dictionary in Statistical Machine Translation	Jul 1, 2015	Domain AdaptationLanguage Modelling	—Unverified	0
Multi-system machine translation using online APIs for English-Latvian	Jul 1, 2015	Language ModellingMachine Translation	CodeCode Available	0
QCRI@QALB-2015 Shared Task: Correction of Arabic Text for Native and Non-Native Speakers' Errors	Jul 1, 2015	Language Modelling	—Unverified	0
QCMUQ@QALB-2015 Shared Task: Combining Character level MT and Error-tolerant Finite-State Recognition for Arabic Spelling Correction	Jul 1, 2015	Language ModellingMachine Translation	—Unverified	0
Toward Tweets Normalization Using Maximum Entropy	Jul 1, 2015	Language Modelling	—Unverified	0
Neural Network Transduction Models in Transliteration Generation	Jul 1, 2015	Language ModellingMachine Translation	—Unverified	0
Dependency Parsing with Graph Rewriting	Jul 1, 2015	Dependency ParsingLanguage Modelling	—Unverified	0
Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length	Jul 1, 2015	Language ModellingMachine Translation	—Unverified	0
Incremental Recurrent Neural Network Dependency Parser with Search-based Discriminative Training	Jul 1, 2015	Dependency ParsingFeature Engineering	—Unverified	0
Instance Selection Improves Cross-Lingual Model Training for Fine-Grained Sentiment Analysis	Jul 1, 2015	Language ModellingMachine Translation	—Unverified	0
Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction	Jul 1, 2015	Language ModellingRepresentation Learning	—Unverified	0
Tibetan Unknown Word Identification from News Corpora for Supporting Lexicon-based Tibetan Word Segmentation	Jul 1, 2015	Language Modelling	—Unverified	0
Reducing infrequent-token perplexity via variational corpora	Jul 1, 2015	Language Modelling	—Unverified	0

Show:10 25 50

← PrevPage 339 of 353Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified