SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Showing 33013350 of 17610 papers

TitleStatusHype
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and GenerationCode1
AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language ModelsCode1
Knowledge-Augmented Language Models for Cause-Effect Relation ClassificationCode1
Self-Supervised Learning for speech recognition with Intermediate layer supervisionCode1
Efficient Hierarchical Domain Adaptation for Pretrained Language ModelsCode1
UNIREX: A Unified Learning Framework for Language Model Rationale ExtractionCode1
Learning To Retrieve Prompts for In-Context LearningCode1
Value Retrieval with Arbitrary Queries for Form-like DocumentsCode1
Improving Conversational Recommendation Systems' Quality with Context-Aware Item Meta InformationCode1
SPTS: Single-Point Text SpottingCode1
Deciphering antibody affinity maturation with language models and weakly supervised learningCode1
Step-unrolled Denoising Autoencoders for Text GenerationCode1
MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based FinetuningCode1
Zero-Shot Recommendation as Language ModelingCode1
Prompting Visual-Language Models for Efficient Video UnderstandingCode1
MLP Architectures for Vision-and-Language Modeling: An Empirical StudyCode1
Quantifying Adaptability in Pre-trained Language Models with 500 TasksCode1
Keeping it Simple: Language Models can learn Complex Molecular DistributionsCode1
Causal Distillation for Language ModelsCode1
InfoLM: A New Metric to Evaluate Summarization & Data2Text GenerationCode1
DenseCLIP: Language-Guided Dense Prediction with Context-Aware PromptingCode1
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network ModelsCode1
A Simple Long-Tailed Recognition Baseline via Vision-Language ModelCode1
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic ArithmeticCode1
Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language ModelCode1
Evaluating the Robustness of Retrieval Pipelines with Query Variation GeneratorsCode1
Romanian Speech Recognition Experiments from the ROBIN ProjectCode1
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language ModelingCode1
Enhancing Multilingual Language Model with Massive Multilingual Knowledge TriplesCode1
RoBERTuito: a pre-trained language model for social media text in SpanishCode1
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding MatchingCode1
Interpreting Language Models Through Knowledge Graph ExtractionCode1
iBOT: Image BERT Pre-Training with Online TokenizerCode1
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient FrameworkCode1
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language ModelingCode1
Lexically Aware Semi-Supervised Learning for OCR Post-CorrectionCode1
The neural architecture of language: Integrative modeling converges on predictive processingCode1
The Klarna Product Page Dataset: Web Element Nomination with Graph Neural Networks and Large Language ModelsCode1
An Explanation of In-context Learning as Implicit Bayesian InferenceCode1
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak DecoderCode1
ParsiNorm: A Persian Toolkit for Speech Processing NormalizationCode1
A Model of Cross-Lingual Knowledge-Grounded Response Generation for Open-Domain Dialogue SystemsCode1
TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding LearningCode1
AttentionRank: Unsupervised Keyphrase Extraction using Self and Cross AttentionsCode1
Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced LanguagesCode1
With a Little Help from my Temporal Context: Multimodal Egocentric Action RecognitionCode1
Effective Use of Graph Convolution Network and Contextual Sub-Tree for Commodity News Event ExtractionCode1
A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval GreekCode1
AESOP: Paraphrase Generation with Adaptive Syntactic ControlCode1
DILBERT: Customized Pre-Training for Domain Adaptation with Category Shift, with an Application to Aspect ExtractionCode1
Show:102550
← PrevPage 67 of 353Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67Unverified
2GRUValidation perplexity53.78Unverified
3LSTMValidation perplexity52.73Unverified
4LSTMTest perplexity48.7Unverified
5Temporal CNNTest perplexity45.2Unverified
6TCNTest perplexity45.19Unverified
7GCNN-8Test perplexity44.9Unverified
8Neural cache model (size = 100)Test perplexity44.8Unverified
9Neural cache model (size = 2,000)Test perplexity40.8Unverified
10GPT-2 SmallTest perplexity37.5Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47Unverified
2Seq-U-NetTest perplexity107.95Unverified
3GRU (Bai et al., 2018)Test perplexity92.48Unverified
4R-TransformerTest perplexity84.38Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67Unverified
2HypernetworksBit per Character (BPC)1.34Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33Unverified
4LN HM-LSTMBit per Character (BPC)1.32Unverified
5ByteNetBit per Character (BPC)1.31Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25Unverified
8Large mLSTMBit per Character (BPC)1.24Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33Unverified
2OPT 125MTest perplexity32.26Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1Unverified
4OPT 1.3BTest perplexity19.55Unverified
5GPT-Neo 125MTest perplexity17.83Unverified
6OPT 2.7BTest perplexity17.81Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12Unverified
8GPT-Neo 1.3BTest perplexity11.46Unverified
9Transformer 125MTest perplexity10.7Unverified
10GPT-Neo 2.7BTest perplexity10.44Unverified