SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Showing 1115111200 of 17610 papers

TitleStatusHype
Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training0
Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models0
Like a bilingual baby: The advantage of visually grounding a bilingual language model0
Can Language Models Be Specific? How?Code0
Continual Training of Language Models for Few-Shot LearningCode2
A Kernel-Based View of Language Model Fine-TuningCode1
Instance Regularization for Discriminative Language Model Pre-trainingCode0
Understanding the Failure of Batch Normalization for Transformers in NLPCode1
Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization0
The Minimum Wage as an Anchor: Effects on Determinations of Fairness by Humans and AI0
Leveraging Key Information Modeling to Improve Less-Data Constrained News Headline Generation via Duality Fine-Tuning0
Readability Controllable Biomedical Document Summarization0
CORE: A Retrieve-then-Edit Framework for Counterfactual Data GenerationCode0
Do Children Texts Hold The Key To Commonsense Knowledge?0
Bridging CLIP and StyleGAN through Latent Alignment for Image Editing0
Scaling Up Probabilistic Circuits by Latent Variable DistillationCode0
QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation0
Cross-Align: Modeling Deep Cross-lingual Interactions for Word AlignmentCode1
Better Pre-Training by Reducing Representation Confusion0
Controllable Dialogue Simulation with In-Context LearningCode1
InfoCSE: Information-aggregated Contrastive Learning of Sentence EmbeddingsCode1
AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models0
KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document UnderstandingCode0
Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal ModelingCode1
Using Interventions to Improve Out-of-Distribution Generalization of Text-Matching Recommendation Systems0
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal ShiftsCode2
Large Language Models can Implement Policy Iteration0
Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context ExpertsCode0
UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentationCode0
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language UnderstandingCode2
Novice Type Error Diagnosis with Natural Language Models0
Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models0
PQLM -- Multilingual Decentralized Portable Quantum Language Model for Privacy Protection0
Improving Large-scale Paraphrase Acquisition and Generation0
BootAug: Boosting Text Augmentation via Hybrid Instance Filtering FrameworkCode1
Binding Language Models in Symbolic LanguagesCode2
Generative Entity Typing with Curriculum LearningCode1
Conversational Semantic Role Labeling with Predicate-Oriented Latent Graph0
Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot LearnersCode1
Improving the Sample Efficiency of Prompt Tuning with Domain AdaptationCode0
Just ClozE! A Novel Framework for Evaluating the Factual Consistency Faster in Abstractive SummarizationCode0
Vision Transformer Based Model for Describing a Set of Images as a Story0
VIMA: General Robot Manipulation with Multimodal PromptsCode2
Reprogramming Pretrained Language Models for Antibody Sequence InfillingCode1
Large Language Models are Pretty Good Zero-Shot Video Game Bug DetectorsCode1
Revisiting Syllables in Language Modelling and their Application on Low-Resource Machine Translation0
GLM-130B: An Open Bilingual Pre-trained ModelCode6
Nonparametric Decoding for Generative RetrievalCode1
Antibody Representation Learning for Drug Discovery0
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model0
Show:102550
← PrevPage 224 of 353Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67Unverified
2GRUValidation perplexity53.78Unverified
3LSTMValidation perplexity52.73Unverified
4LSTMTest perplexity48.7Unverified
5Temporal CNNTest perplexity45.2Unverified
6TCNTest perplexity45.19Unverified
7GCNN-8Test perplexity44.9Unverified
8Neural cache model (size = 100)Test perplexity44.8Unverified
9Neural cache model (size = 2,000)Test perplexity40.8Unverified
10GPT-2 SmallTest perplexity37.5Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47Unverified
2Seq-U-NetTest perplexity107.95Unverified
3GRU (Bai et al., 2018)Test perplexity92.48Unverified
4R-TransformerTest perplexity84.38Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67Unverified
2HypernetworksBit per Character (BPC)1.34Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33Unverified
4LN HM-LSTMBit per Character (BPC)1.32Unverified
5ByteNetBit per Character (BPC)1.31Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25Unverified
8Large mLSTMBit per Character (BPC)1.24Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33Unverified
2OPT 125MTest perplexity32.26Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1Unverified
4OPT 1.3BTest perplexity19.55Unverified
5GPT-Neo 125MTest perplexity17.83Unverified
6OPT 2.7BTest perplexity17.81Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12Unverified
8GPT-Neo 1.3BTest perplexity11.46Unverified
9Transformer 125MTest perplexity10.7Unverified
10GPT-Neo 2.7BTest perplexity10.44Unverified