SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 9951–10000 of 17610 papers

TitleStatusHype
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex—0
On Sampling-Based Training Criteria for Neural Language Modeling—0
On Scaling Up a Multilingual Vision and Language Model—0
On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research—0
ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model—0
On Speculative Decoding for Multimodal Large Language Models—0
On Speeding Up Language Model Evaluation—0
On Text Style Transfer via Style Masked Language Models—0
Making Large Language Models Better Reasoners with Step-Aware Verifier—0
Towards Federated RLHF with Aggregated Client Preference for LLMs—0
On the comparability of Pre-trained Language Models—0
On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation—0
On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation—0
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent—0
On the Computational Modelling of Michif Verbal Morphology—0
On the Discussion of Large Language Models: Symmetry of Agents and Interplay with Prompts—0
On the Effectiveness of Acoustic BPE in Decoder-Only TTS—0
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation—0
On the Effectiveness of Neural Text Generation based Data Augmentation for Recognition of Morphologically Rich Speech—0
On the Effectiveness of Pretrained Models for API Learning—0
On the Effectiveness of Using Syntactic and Shallow Semantic Tree Kernels for Automatic Assessment of Essays—0
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model—0
On the Effect of Word Order on Cross-lingual Sentiment Analysis—0
On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models—0
On the Exploration of English to Urdu Machine Translation—0
On-the-Fly Attention Modulation for Neural Generation—0
On-the-fly Text Retrieval for End-to-End ASR Adaptation—0
On the importance of Data Scale in Pretraining Arabic Language Models—0
On the importance of pre-training data volume for compact language models—0
On the Importance of Text Preprocessing for Multimodal Representation Learning and Pathology Report Generation—0
On the Influence of Masking Policies in Intermediate Pre-training—0
On the Limitations of Steering in Language Model Alignment—0
On the limit of English conversational speech recognition—0
On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse—0
On the Multilingual Capabilities of Very Large-Scale English Language Models—0
On the N-gram Approximation of Pre-trained Language Models—0
On the Origins of Linear Representations in Large Language Models—0
On the Planning, Search, and Memorization Capabilities of Large Language Models—0
On the Power of Convolution Augmented Transformer—0
On the Power of Decision Trees in Auto-Regressive Language Modeling—0
On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descent—0
On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers—0
On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling—0
On the Representation Collapse of Sparse Mixture of Experts—0
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling—0
On the Role of Bidirectionality in Language Model Pre-Training—0
On the Role of Corpus Ordering in Language Modeling—0
On the Same Page: Dimensions of Perceived Shared Understanding in Human-AI Interaction—0
On the Scalability of GNNs for Molecular Graphs—0
Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learning—0
Show:102550
← PrevPage 200 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified