SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Showing 1345113500 of 17610 papers

TitleStatusHype
Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision0
Modeling Mathematical Notation Semantics in Academic Papers0
Small Model and In-Domain Data Are All You Need0
Novel Natural Language Summarization of Program Code via Leveraging Multiple Input Representations0
NICT Kyoto Submission for the WMT’21 Quality Estimation Task: Multimetric Multilingual Pretraining for Critical Error Detection0
Japanese Zero Anaphora Resolution Can Benefit from Parallel Texts Through Neural Transfer Learning0
Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units0
R-BERT-CNN: Drug-target interactions extraction from biomedical literature0
PnPOOD : Out-Of-Distribution Detection for Text Classification via Plug andPlay Data Augmentation0
Automatic Knowledge Augmentation for Generative Commonsense Reasoning0
EmpBot: A T5-based Empathetic Chatbot focusing on Sentiments0
Combining Unsupervised and Text Augmented Semi-Supervised Learning for Low Resourced Autoregressive Speech Recognition0
Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model0
Node Feature Extraction by Self-Supervised Multi-scale Neighborhood PredictionCode0
Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-TuningCode0
Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?0
Distributionally Robust Recurrent Decoders with Random Network Distillation0
Paradigm Shift in Language Modeling: Revisiting CNN for Modeling Sanskrit Originated Bengali and Hindi Language0
No News is Good News: A Critique of the One Billion Word Benchmark0
Sentence Punctuation for Collaborative Commentary Generation in Esports Live-Streaming0
Text Counterfactuals via Latent Optimization and Shapley-Guided SearchCode0
Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach0
JavaBERT: Training a transformer-based model for the Java programming languageCode0
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training0
Knowledge Graph informed Fake News Classification via Heterogeneous Representation EnsemblesCode0
Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair PredictionCode0
DEEPAGÉ: Answering Questions in Portuguese about the Brazilian EnvironmentCode0
Automatic Learning of Subword Dependent Model Scales0
NormFormer: Improved Transformer Pretraining with Extra Normalization0
Reminding the Incremental Language Model via Data-Free Self-Distillation0
Deconfounded and Explainable Interactive Vision-Language Retrieval of Complex Scenes0
ASR4REAL: An extended benchmark for speech models0
Echo-Attention: Attend Once and Get N Attentions for Free0
DEMix Layers: Disentangling Domains for Modular Language Modeling0
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model CompressionCode0
A Novel Metric for Evaluating Semantics PreservationCode0
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models0
Leveraging Knowledge in Multilingual Commonsense Reasoning0
Sharpness-Aware Minimization Improves Language Model Generalization0
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora0
Prix-LM: Pretraining for Multilingual Knowledge Base ConstructionCode0
Multilingual unsupervised sequence segmentation transfers to extremely low-resource languagesCode0
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens0
N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking0
On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation0
xGQA: Cross-Lingual Visual Question Answering0
A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification0
Intent-based Product Collections for E-commerce using Pretrained Language Models0
Crisis Domain Adaptation Using Sequence-to-sequence TransformersCode0
DS-TOD: Efficient Domain Specialization for Task Oriented DialogCode0
Show:102550
← PrevPage 270 of 353Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67Unverified
2GRUValidation perplexity53.78Unverified
3LSTMValidation perplexity52.73Unverified
4LSTMTest perplexity48.7Unverified
5Temporal CNNTest perplexity45.2Unverified
6TCNTest perplexity45.19Unverified
7GCNN-8Test perplexity44.9Unverified
8Neural cache model (size = 100)Test perplexity44.8Unverified
9Neural cache model (size = 2,000)Test perplexity40.8Unverified
10GPT-2 SmallTest perplexity37.5Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47Unverified
2Seq-U-NetTest perplexity107.95Unverified
3GRU (Bai et al., 2018)Test perplexity92.48Unverified
4R-TransformerTest perplexity84.38Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67Unverified
2HypernetworksBit per Character (BPC)1.34Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33Unverified
4LN HM-LSTMBit per Character (BPC)1.32Unverified
5ByteNetBit per Character (BPC)1.31Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25Unverified
8Large mLSTMBit per Character (BPC)1.24Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33Unverified
2OPT 125MTest perplexity32.26Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1Unverified
4OPT 1.3BTest perplexity19.55Unverified
5GPT-Neo 125MTest perplexity17.83Unverified
6OPT 2.7BTest perplexity17.81Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12Unverified
8GPT-Neo 1.3BTest perplexity11.46Unverified
9Transformer 125MTest perplexity10.7Unverified
10GPT-Neo 2.7BTest perplexity10.44Unverified