SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Showing 1195112000 of 17610 papers

TitleStatusHype
The Causal News Corpus: Annotating Causal Relations in Event Sentences from NewsCode1
Which Discriminator for Cooperative Text Generation?Code1
Emotion-Aware Transformer Encoder for Empathetic Dialogue GenerationCode1
Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability0
Taygete at SemEval-2022 Task 4: RoBERTa based models for detecting Patronising and Condescending Language0
Locally Aggregated Feature Attribution on Natural Language Model Understanding0
Sparse and Dense Approaches for the Full-rank Retrieval of Responses for DialoguesCode1
KALA: Knowledge-Augmented Language Model AdaptationCode1
WaBERT: A Low-resource End-to-end Model for Spoken Language Understanding and Speech-to-BERT Alignment0
DiffCSE: Difference-based Contrastive Learning for Sentence EmbeddingsCode2
Making the Most of Text Semantics to Improve Biomedical Vision--Language ProcessingCode0
On the Representation Collapse of Sparse Mixture of Experts0
When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout ProbesCode0
Detecting Unintended Memorization in Language-Model-Fused ASR0
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks0
DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue GenerationCode1
Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion0
Context-Aware Language Modeling for Goal-Oriented Dialogue Systems0
A Study on Prompt-based Few-Shot Learning Methods for Belief State Tracking in Task-oriented Dialog Systems0
L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language ModelsCode1
StableMoE: Stable Routing Strategy for Mixture of ExpertsCode1
LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingCode0
Zero-shot Entity and Tweet Characterization with Designed Conditional Prompts and Contexts0
UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language0
WordAlchemy: A transformer-based Reverse Dictionary0
A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment AnalysisCode1
BLCU-ICALL at SemEval-2022 Task 1: Cross-Attention Multitasking Framework for Definition ModelingCode0
Contrastive Learning with Hard Negative Entities for Entity Set ExpansionCode1
SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words0
Is Surprisal in Issue Trackers Actionable?0
Text Revision by On-the-Fly Representation OptimizationCode0
LaMemo: Language Modeling with Look-Ahead MemoryCode0
Improving Passage Retrieval with Zero-Shot Question GenerationCode1
Evaluation Benchmarks for Spanish Sentence RepresentationsCode1
Generative power of a protein language model trained on multiple sequence alignmentsCode1
GPT-NeoX-20B: An Open-Source Autoregressive Language ModelCode1
Rows from Many Sources: Enriching row completions from Wikidata with a pre-trained Language Model0
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-TuningCode1
HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection0
Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot ClassificationCode1
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding0
GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text GenerationCode1
Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change0
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?Code1
Mining Logical Event Schemas From Pre-Trained Language Models0
A Generative Language Model for Few-shot Aspect-Based Sentiment AnalysisCode1
Adapting BigScience Multilingual Model to Unseen Languages0
Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling0
Breaking Character: Are Subwords Good Enough for MRLs After All?0
Pushing on Personality Detection from Verbal Behavior: A Transformer Meets Text Contours of Psycholinguistic Features0
Show:102550
← PrevPage 240 of 353Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67Unverified
2GRUValidation perplexity53.78Unverified
3LSTMValidation perplexity52.73Unverified
4LSTMTest perplexity48.7Unverified
5Temporal CNNTest perplexity45.2Unverified
6TCNTest perplexity45.19Unverified
7GCNN-8Test perplexity44.9Unverified
8Neural cache model (size = 100)Test perplexity44.8Unverified
9Neural cache model (size = 2,000)Test perplexity40.8Unverified
10GPT-2 SmallTest perplexity37.5Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47Unverified
2Seq-U-NetTest perplexity107.95Unverified
3GRU (Bai et al., 2018)Test perplexity92.48Unverified
4R-TransformerTest perplexity84.38Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67Unverified
2HypernetworksBit per Character (BPC)1.34Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33Unverified
4LN HM-LSTMBit per Character (BPC)1.32Unverified
5ByteNetBit per Character (BPC)1.31Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25Unverified
8Large mLSTMBit per Character (BPC)1.24Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33Unverified
2OPT 125MTest perplexity32.26Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1Unverified
4OPT 1.3BTest perplexity19.55Unverified
5GPT-Neo 125MTest perplexity17.83Unverified
6OPT 2.7BTest perplexity17.81Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12Unverified
8GPT-Neo 1.3BTest perplexity11.46Unverified
9Transformer 125MTest perplexity10.7Unverified
10GPT-Neo 2.7BTest perplexity10.44Unverified