SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 12851–12900 of 17610 papers

TitleStatusHype
MTL-SLT: Multi-Task Learning for Spoken Language Tasks—0
Leveraging Similar Users for Personalized Language Modeling with Limited Data—0
To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers—0
LayoutBERT: Masked Language Layout Model for Object Insertion—0
Self-Programming Artificial Intelligence Using Code-Generating Language Models—0
Visualizing and Explaining Language Models—0
Vision-Language Pre-Training for Boosting Scene Text DetectorsCode0
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining—0
Training Language Models with Language Feedback—0
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model—0
UBERT: A Novel Language Model for Synonymy Prediction at Scale in the UMLS MetathesaurusCode0
RigoBERTa: A State-of-the-Art Language Model For Spanish—0
Probing Simile Knowledge from Pre-trained Language ModelsCode0
A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical TransformerCode0
Efficient Machine Translation Domain AdaptationCode0
You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private PersonasCode0
Parkinson's disease diagnostics using AI and natural language knowledge transfer—0
Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors—0
Super-Prompting: Utilizing Model-Independent Contextual Data to Reduce Data Annotation Required in Visual Commonsense Tasks—0
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference—0
Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials—0
C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval—0
Unsupervised Representation Learning of Player Behavioral Data with Confidence Guided MaskingCode0
WaBERT: A Low-resource End-to-end Model for Spoken Language Understanding and Speech-to-BERT Alignment—0
Locally Aggregated Feature Attribution on Natural Language Model Understanding—0
Taygete at SemEval-2022 Task 4: RoBERTa based models for detecting Patronising and Condescending Language—0
Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability—0
Making the Most of Text Semantics to Improve Biomedical Vision--Language ProcessingCode0
On the Representation Collapse of Sparse Mixture of Experts—0
When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout ProbesCode0
Detecting Unintended Memorization in Language-Model-Fused ASR—0
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks—0
Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion—0
UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language—0
LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingCode0
Context-Aware Language Modeling for Goal-Oriented Dialogue Systems—0
A Study on Prompt-based Few-Shot Learning Methods for Belief State Tracking in Task-oriented Dialog Systems—0
Zero-shot Entity and Tweet Characterization with Designed Conditional Prompts and Contexts—0
WordAlchemy: A transformer-based Reverse Dictionary—0
SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words—0
BLCU-ICALL at SemEval-2022 Task 1: Cross-Attention Multitasking Framework for Definition ModelingCode0
Is Surprisal in Issue Trackers Actionable?—0
LaMemo: Language Modeling with Look-Ahead MemoryCode0
Text Revision by On-the-Fly Representation OptimizationCode0
Rows from Many Sources: Enriching row completions from Wikidata with a pre-trained Language Model—0
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding—0
HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection—0
Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change—0
Mining Logical Event Schemas From Pre-Trained Language Models—0
Adapting BigScience Multilingual Model to Unseen Languages—0
Show:102550
← PrevPage 258 of 353Next →
All datasetsWikiText-103Penn Treebank (Word Level)enwik8The PileWikiText-2LAMBADAOne Billion WordText8Penn Treebank (Character Level)Hutter PrizeOpenWebTextSALMon

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67—Unverified
2GRUValidation perplexity53.78—Unverified
3LSTMValidation perplexity52.73—Unverified
4LSTMTest perplexity48.7—Unverified
5Temporal CNNTest perplexity45.2—Unverified
6TCNTest perplexity45.19—Unverified
7GCNN-8Test perplexity44.9—Unverified
8Neural cache model (size = 100)Test perplexity44.8—Unverified
9Neural cache model (size = 2,000)Test perplexity40.8—Unverified
10GPT-2 SmallTest perplexity37.5—Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47—Unverified
2Seq-U-NetTest perplexity107.95—Unverified
3GRU (Bai et al., 2018)Test perplexity92.48—Unverified
4R-TransformerTest perplexity84.38—Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7—Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7—Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93—Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4—Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2—Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66—Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67—Unverified
2HypernetworksBit per Character (BPC)1.34—Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33—Unverified
4LN HM-LSTMBit per Character (BPC)1.32—Unverified
5ByteNetBit per Character (BPC)1.31—Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27—Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25—Unverified
8Large mLSTMBit per Character (BPC)1.24—Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23—Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22—Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33—Unverified
2OPT 125MTest perplexity32.26—Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1—Unverified
4OPT 1.3BTest perplexity19.55—Unverified
5GPT-Neo 125MTest perplexity17.83—Unverified
6OPT 2.7BTest perplexity17.81—Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12—Unverified
8GPT-Neo 1.3BTest perplexity11.46—Unverified
9Transformer 125MTest perplexity10.7—Unverified
10GPT-Neo 2.7BTest perplexity10.44—Unverified