SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 10701–10750 of 17610 papers

TitleStatusHype
TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven OptimizationCode1
Mu^2SLAM: Multitask, Multilingual Speech and Language Models—0
PromptBoosting: Black-Box Text Classification with Ten Forward PassesCode1
Natural Language to Code Generation in Interactive Data Science Notebooks—0
MANTIS at TSAR-2022 Shared Task: Improved Unsupervised Lexical Simplification with Pretrained Encoders—0
KNIFE: Distilling Reasoning Knowledge From Free-Text Rationales—0
Large Language Models are Better Reasoners with Self-VerificationCode1
Reasoning with Language Model Prompting: A SurveyCode3
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering—0
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning—0
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot PromptingCode1
Discovering Language Model Behaviors with Model-Written EvaluationsCode3
Emergent Analogical Reasoning in Large Language ModelsCode1
Unnatural Instructions: Tuning Language Models with (Almost) No Human LaborCode1
Very Large Language Model as a Unified Methodology of Text MiningCode0
Language model acceptability judgements are not always robust to context—0
Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language ModelCode1
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion ScaleCode1
Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing—0
Claim Optimization in Computational ArgumentationCode0
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation PerturbationCode1
POIBERT: A Transformer-based Model for the Tour Recommendation Problem—0
LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension—0
Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language—0
ALERT: Adapting Language Models to Reasoning Tasks—0
Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-GenerationCode1
Homonymy Information for English WordNetCode0
Attention as a Guide for Simultaneous Speech TranslationCode0
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference—0
Improving Chess Commentaries by Combining Language Models with Symbolic Reasoning Engines—0
Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability DetectionCode1
Efficient Pre-training of Masked Language Model via Concept-based Curriculum MaskingCode1
Efficient Long Sequence Modeling via State Space Augmented TransformerCode1
The Effects of In-domain Corpus Size on pre-training BERTCode0
On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot ReasoningCode1
Joint processing of linguistic properties in brains and language modelsCode0
CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models—0
DeepJoin: Joinable Table Discovery with Pre-trained Language Models—0
Enhancing Indic Handwritten Text Recognition Using Global Semantic InformationCode1
CLIPPO: Image-and-Language Understanding from Pixels Only—0
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning—0
MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling—0
The Challenges of HTR Model Training: Feedback from the Project Donner le gout de l'archive a l'ere numerique—0
Technical Report -- Competition Solution for Prompt Tuning using Pretrained Language Model—0
What do Vision Transformers Learn? A Visual ExplorationCode1
Deep Image Style Transfer from Freeform Text—0
Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?Code0
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming LanguagesCode6
CNO-LSTM: A Chaotic Neural Oscillatory Long Short-Term Memory Model for Text Classification—0
DexBERT: Effective, Task-Agnostic and Fine-grained Representation Learning of Android BytecodeCode1
Show:102550
← PrevPage 215 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified