SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 14401–14450 of 17610 papers

TitleStatusHype
Joint Extraction of Entity and Relation with Information Redundancy Elimination—0
Automated Coding of Under-Studied Medical Concept Domains: Linking Physical Activity Reports to the International Classification of Functioning, Disability, and HealthCode0
Automatic coding of students' writing via Contrastive Representation Learning in the Wasserstein space—0
Unigram-Normalized Perplexity as a Language Model Performance Measure with Different Vocabulary Sizes—0
Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo ApproachCode0
Adam^+: A Stochastic Method with Adaptive Variance Reduction—0
Does BERT Understand Sentiment? Leveraging Comparisons Between Contextual and Non-Contextual Embeddings to Improve Aspect-Based Sentiment Models—0
Multi-task Language Modeling for Improving Speech Recognition of Rare Words—0
Self-Supervised learning with cross-modal transformers for emotion recognition—0
Collaborative Storytelling with Large-scale Neural Language Models—0
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural NetworksCode0
A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus—0
Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification—0
Predictions For Pre-training Language Models—0
Structural and Functional Decomposition for Personality Image Captioning in a Communication Game—0
MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining—0
Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter—0
NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge BasesCode0
Conditioned Natural Language Generation using only Unconditioned Language Model: An Exploration—0
Re-framing Incremental Deep Language Models for Dialogue Processing with Multi-task LearningCode0
Exploring the Value of Personalized Word Embeddings—0
E.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks—0
Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration—0
Positional Artefacts Propagate Through Masked Language Model Embeddings—0
Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources—0
Language Model is All You Need: Natural Language Understanding as Question Answering—0
A Neuro-Symbolic Method for Solving Differential and Functional Equations—0
Indic-Transformers: An Analysis of Transformer Language Models for Indian LanguagesCode0
Improving RNN transducer with normalized jointer network—0
Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition—0
Sound Natural: Content Rephrasing in Dialog SystemsCode0
Modeling Event Salience in Narratives via Barthes' Cardinal Functions—0
An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution—0
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP—0
Improving Variational Autoencoder for Text Modelling with Timestep-Wise RegularisationCode0
A3-108 Machine Translation System for Similar Language Translation Shared Task 2020—0
Coding Textual Inputs Boosts the Accuracy of Neural NetworksCode0
Biomedical Event Extraction as Multi-turn Question AnsweringCode0
CXP949 at WNUT-2020 Task 2: Extracting Informative COVID-19 Tweets - RoBERTa Ensembles and The Continued Relevance of Handcrafted Features—0
An Empirical Exploration of Local Ordering Pre-training for Structured Prediction—0
Explainable Clinical Decision Support from Text—0
Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling—0
Controlling the Imprint of Passivization and Negation in Contextualized RepresentationsCode0
Grammaticality and Language Modelling—0
HLTRI at W-NUT 2020 Shared Task-3: COVID-19 Event Extraction from Twitter Using Multi-Task Hopfield Pooling—0
A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning—0
``I'd rather just go to bed'': Understanding Indirect Answers—0
Alibaba’s Submission for the WMT 2020 APE Shared Task: Improving Automatic Post-Editing with Pre-trained Conditional Cross-Lingual BERT—0
ComplexDataLab at W-NUT 2020 Task 2: Detecting Informative COVID-19 Tweets by Attending over Linked Documents—0
imec-ETRO-VUB at W-NUT 2020 Shared Task-3: A multilabel BERT-based system for predicting COVID-19 events—0
Show:102550
← PrevPage 289 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified