SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 15751–15800 of 17610 papers

TitleStatusHype
Language Models Learn POS First—0
Representation of Word Meaning in the Intermediate Projection Layer of a Neural Language Model—0
Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis—0
What do RNN Language Models Learn about Filler--Gap Dependencies?—0
Juman++: A Morphological Analysis Toolkit for Scriptio ContinuaCode0
On the End-to-End Solution to Mandarin-English Code-switching Speech RecognitionCode0
Understanding Learning Dynamics Of Language Models with SVCCA—0
Towards Coherent and Cohesive Long-form Text Generation—0
Improving Machine Reading Comprehension with General Reading StrategiesCode0
Towards End-to-end Automatic Code-Switching Speech Recognition—0
Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model—0
Recurrent Attention Unit—0
Language Modeling with Sparse Product of Sememe ExpertsCode0
Visual Re-ranking with Natural Language Understanding for Text SpottingCode0
Counting in Language with RNNs—0
Cascaded CNN-resBiLSTM-CTC: An End-to-End Acoustic Model For Speech Recognition—0
Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative TrainingCode0
Can Entropy Explain Successor Surprisal Effects in Reading?—0
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary CellsCode1
Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling—0
Universal Language Model Fine-Tuning with Subword Tokenization for PolishCode0
A Deep Generative Acoustic Model for Compositional Automatic Speech Recognition—0
Neural Transition-based Syntactic Linearization—0
Language Modeling at Scale—0
Training Neural Speech Recognition Systems with Synthetic Speech Augmentation—0
Bridging HMMs and RNNs through Architectural Transformations—0
Ordered Neurons: Integrating Tree Structures into Recurrent Neural NetworksCode0
Real-time Neural-based Input Method—0
Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed RepresentationsCode0
Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural NetworksCode0
Structured Content Preservation for Unsupervised Text Style TransferCode0
Trellis Networks for Sequence ModelingCode0
Exploring the Use of Attention within an Neural Machine Translation Decoder States to Translate Idioms—0
textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional PriorCode0
Unsupervised Neural Word Segmentation for Chinese via Segmental Language ModelingCode0
Understanding Recurrent Neural Architectures by Analyzing and Synthesizing Long Distance Dependencies in Benchmark Sequential Datasets—0
Learning Compressed Transforms with Low Displacement RankCode0
Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling—0
The Sogou-TIIC Speech Translation System for IWSLT 2018—0
Zero-Shot Learning Based Approach For Medieval Word Recognition Using Deep-Learned Features—0
會議語音辨識使用語者資訊之語言模型調適技術 (On the Use of Speaker-Aware Language Model Adaptation Techniques for Meeting Speech Recognition ) [In Chinese]—0
Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared taskCode1
Supervised and Unsupervised Minimalist Quality Estimators: Vicomtech's Participation in the WMT 2018 Quality Estimation Task—0
The ILSP/ARC submission to the WMT 2018 Parallel Corpus Filtering Shared Task—0
The LMU Munich Unsupervised Machine Translation Systems—0
Learning Comment Controversy Prediction in Web Discussions Using Incidentally Supervised Multi-Task CNNs—0
Keep It or Not: Word Level Quality Estimation for Post-Editing—0
RTM results for Predicting Translation Performance—0
The JHU Parallel Corpus Filtering Systems for WMT 2018—0
Sounds Wilde. Phonetically Extended Embeddings for Author-Stylized Poetry Generation—0
Show:102550
← PrevPage 316 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified