SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 14951–15000 of 17610 papers

TitleStatusHype
Efficient long-distance relation extraction with DG-SpanBERT—0
Exploring Early Prediction of Buyer-Seller Negotiation Outcomes—0
Adding A Filter Based on The Discriminator to Improve Unconditional Text GenerationCode0
Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2Code0
Semi-supervised acoustic and language model training for English-isiZulu code-switched speech recognition—0
Syntax-driven Iterative Expansion Language Models for Controllable Text Generation—0
Recognizing Long Grammatical Sequences Using Recurrent Networks Augmented With An External Differentiable Stack—0
CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection—0
Adversarial Transfer Learning for Punctuation Restoration—0
Towards Productionizing Subjective Search Systems—0
NukeBERT: A Pre-trained language model for Low Resource Nuclear DomainCode0
A Hierarchical Transformer for Unsupervised Parsing—0
Abstractive Text Summarization based on Language Model Conditioning and Locality ModelingCode0
Meta Fine-Tuning Neural Language Models for Multi-Domain Text MiningCode0
Common-Knowledge Concept Recognition for SEVACode0
SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection—0
Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning—0
TNT-KID: Transformer-based Neural Tagger for Keyword IdentificationCode0
Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies—0
Multi-label natural language processing to identify diagnosis and procedure codes from MIMIC-III inpatient notes—0
Key Phrase Classification in Complex Assignments—0
Finnish Language Modeling with Deep Transformer Models—0
Hybrid Autoregressive Transducer (hat)—0
Meta-CoTGAN: A Meta Cooperative Training Paradigm for Improving Adversarial Text Generation—0
What the [MASK]? Making Sense of Language-Specific BERT Models—0
XGPT: Cross-modal Generative Pre-Training for Image Captioning—0
Improving Uyghur ASR systems with decoders using morpheme-based language models—0
Meta-Embeddings Based On Self-Attention—0
A Deep Generative Model for Fragment-Based Molecule GenerationCode0
Using a thousand optimization tasks to learn hyperparameter search strategies—0
A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition—0
Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units—0
Sparse Sinkhorn AttentionCode0
Object Relational Graph with Teacher-Recommended Learning for Video Captioning—0
Quantized Neural Network Inference with Precision Batching—0
A more abstractive summarization model—0
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity—0
Sequence Preserving Network Traffic Generation—0
MaxUp: A Simple Way to Improve Generalization of Neural Network TrainingCode0
Scalable Second Order Optimization for Deep LearningCode0
A Systematic Comparison of Architectures for Document-Level Sentiment ClassificationCode0
Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents—0
A Financial Service Chatbot based on Deep Bidirectional Transformers—0
Global and Local Feature Learning for Ego-Network Analysis—0
FQuAD: French Question Answering Dataset—0
A Data Efficient End-To-End Spoken Language Understanding Architecture—0
Comparison of Turkish Word Representations Trained on Different Morphological Forms—0
CBAG: Conditional Biomedical Abstract Generation—0
Deep Learning for Source Code Modeling and Generation: Models, Applications and ChallengesCode0
Regularizing activations in neural networks via distribution matching with the Wasserstein metric—0
Show:102550
← PrevPage 300 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified