SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 14551–14600 of 17610 papers

TitleStatusHype
Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation—0
Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering—0
AdaBelief Optimizer: Adapting Stepsizes by theBelief in Observed Gradients—0
QuesBELM: A BERT based Ensemble Language Model for Natural Questions—0
Towards Automatic Sentiment-based Topic Phrase Generation—0
Question Answering over Knowledge Base using Language Model Embeddings—0
Hierarchical Multitask Learning Approach for BERT—0
Consistency and Coherency Enhanced Story Generation—0
Human Adversarial QA: Did the Model Understand the Paragraph?—0
Detecting ESG topics using domain-specific language models and data augmentation approaches—0
Substance over Style: Document-Level Targeted Content TransferCode0
Multi-task Learning of Negation and Speculation for Targeted Sentiment ClassificationCode0
A Transformer Based Pitch Sequence Autoencoder with MIDI Augmentation—0
Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses—0
CXP949 at WNUT-2020 Task 2: Extracting Informative COVID-19 Tweets -- RoBERTa Ensembles and The Continued Relevance of Handcrafted FeaturesCode0
From Language to Language-ish: How Brain-Like is an LSTM's Representation of Nonsensical Language Stimuli?—0
Memformer: A Memory-Augmented Transformer for Sequence Modeling—0
Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion—0
Corruption Is Not All Bad: Incorporating Discourse Structure into Pre-training via Corruption for Essay Scoring—0
Are Some Words Worth More than Others?Code0
Extracting Angina Symptoms from Clinical Notes Using Pre-Trained Transformer Architectures—0
Human-centric Dialog Training via Offline Reinforcement Learning—0
BioMegatron: Larger Biomedical Domain Language Model—0
Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model—0
Multi-Stage Pre-training for Low-Resource Domain Adaptation—0
Pre-trained Language Model Based Active Learning for Sentence Matching—0
Meta-Context Transformers for Domain-Specific Response GenerationCode0
On the Complementary Nature of Knowledge Graph Embedding, Fine Grain Entity Types, and Language ModelingCode0
Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task—0
SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task—0
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLUCode0
Beyond Language: Learning Commonsense from Images for ReasoningCode0
Discourse structure interacts with reference but not syntax in neural language modelsCode0
Latent Tree Learning with Ordered Neurons: What Parses Does It Produce?Code0
Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information MaximizationCode0
Online Back-Parsing for AMR-to-Text GenerationCode0
Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding—0
Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels—0
Solving Historical Dictionary Codes with a Neural Language Model—0
On the importance of pre-training data volume for compact language models—0
Masked ELMo: An evolution of ELMo towards fully contextual RNN language models—0
Tatum-Level Drum Transcription Based on a Convolutional Recurrent Neural Network with Language Model-Based Regularized Training—0
Large Product Key Memory for Pretrained Language ModelsCode0
Precise Task Formalization Matters in Winograd Schema EvaluationsCode0
Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks—0
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks—0
"I'd rather just go to bed": Understanding Indirect Answers—0
Converting the Point of View of Messages Spoken to Virtual AssistantsCode0
Beyond [CLS] through Ranking by Generation—0
Learning to Represent Image and Text with Denotation Graph—0
Show:102550
← PrevPage 292 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified