SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 14901–14950 of 17610 papers

TitleStatusHype
Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks—0
APo-VAE: Text Generation in Hyperbolic Space—0
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTCode0
Semi-Supervised Text Simplification with Back-Translation and Asymmetric Denoising Autoencoders—0
Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning—0
Knowledgeable Dialogue Reading Comprehension on Key Turns—0
Modeling Long Context for Task-Oriented Dialogue State Generation—0
Evaluating Transformer-Based Multilingual Text Classification—0
Expansion via Prediction of Importance with ContextualizationCode0
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning—0
Fast and Memory-Efficient Neural Code Completion—0
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning—0
Challenge Closed-book Science Exam: A Meta-learning Based Question Answering System—0
Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document MatchingCode0
Assessing Discourse Relations in Language Generation from GPT-2—0
Jointly Trained Transformers models for Spoken Language Translation—0
Contextualized Representations Using Textual Encyclopedic Knowledge—0
How fine can fine-tuning be? Learning efficient language models—0
Cross-lingual Information Retrieval with BERTCode0
A Tailored Pre-Training Model for Task-Oriented Dialog GenerationCode0
A Tool for Facilitating OCR Postediting in Historical DocumentsCode0
Coupled intrinsic and extrinsic human language resource-based query expansion—0
On Adversarial Examples for Biomedical NLP Tasks—0
UHH-LT at SemEval-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection—0
QURIOUS: Question Generation Pretraining for Text Generation—0
Keyphrase Prediction With Pre-trained Language Model—0
Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck—0
Fast and Scalable Dialogue State Tracking with Explicit Modular Decomposition—0
Discrete Variational Attention Models for Language Generation—0
Considering Likelihood in NLP Classification Explanations with Occlusion and Language ModelingCode0
Music Generation with Temporal Structure Augmentation—0
On the Encoder-Decoder Incompatibility in Variational Text Modeling and BeyondCode0
Fine-tuning Multi-hop Question Answering with Hierarchical Graph Network—0
Active Sentence Learning by Adversarial Uncertainty Sampling in Discrete Space—0
Do sequence-to-sequence VAEs learn global features of sentences?—0
Entities as Experts: Sparse Memory Access with Entity SupervisionCode0
Extending Text Informativeness Measures to Passage Interestingness Evaluation (Language Model vs. Word Embedding)—0
ControlVAE: Controllable Variational Autoencoder—0
Unified Multi-Criteria Chinese Word Segmentation with BERT—0
Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction—0
Pre-training Text Representations as Meta Learning—0
Detached Error Feedback for Distributed SGD with Random Sparsification—0
Sequence Model Design for Code Completion in the Modern IDE—0
Stacked Convolutional Deep Encoding Network for Video-Text Retrieval—0
Learning to Scale Multilingual Representations for Vision-Language Tasks—0
CALM: Continuous Adaptive Learning for Language Modeling—0
DynaBERT: Dynamic BERT with Adaptive Width and DepthCode0
An investigation of phone-based subword units for end-to-end speech recognition—0
Evaluating Online Continual Learning with CALMCode0
Homophone-based Label Smoothing in End-to-End Automatic Speech Recognition—0
Show:102550
← PrevPage 299 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified