SOTAVerified

Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Showing 1355113600 of 17610 papers

TitleStatusHype
Learning How to Ask: Querying LMs with Mixtures of Soft PromptsCode1
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little0
Large-Scale Self- and Semi-Supervised Learning for Speech Translation0
K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-CommerceCode1
EAT: Enhanced ASR-TTS for Self-supervised Speech RecognitionCode0
Experiments of ASR-based mispronunciation detection for children and adult English learners0
What's in your Head? Emergent Behaviour in Multi-Task Transformer Models0
Transformer-based Methods for Recognizing Ultra Fine-grained Entities (RUFES)0
Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling0
Should Semantic Vector Composition be Explicit? Can it be Linear?0
Large-Scale Contextualised Language Modelling for NorwegianCode1
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question AnsweringCode1
Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation0
Paragraph-level Simplification of Medical TextsCode1
Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models0
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic DependenciesCode1
Building a Swedish Open-Domain Conversational Language ModelCode0
Factual Probing Is [MASK]: Learning vs. Learning to RecallCode1
Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation0
Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architectures0
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt CollectionsCode1
Non-autoregressive Transformer-based End-to-end ASR using BERT0
Lookup-Table Recurrent Language Models for Long Tail Speech Recognition0
Language model fusion for streaming end to end speech recognition0
FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation FrameworkCode0
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMCode0
Extended Parallel Corpus for Amharic-English Machine Translation0
AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application WithCode1
Deep Indexed Active Learning for Matching Heterogeneous Entity RepresentationsCode1
Nutribullets Hybrid: Multi-document Health SummarizationCode0
Revisiting Simple Neural Probabilistic Language ModelsCode1
Pushing the Limits of Non-Autoregressive Speech Recognition0
Interpreting A Pre-trained Model Is A Key For Model Architecture Optimization: A Case Study On Wav2Vec 2.00
Librispeech Transducer Model with Internal Language Model Prior CorrectionCode1
An Empirical Evaluation of Word Embedding Models for Subjectivity Analysis TasksCode0
Communication-Efficient Agnostic Federated Averaging0
LT-LM: a novel non-autoregressive language model for single-shot lattice rescoringCode0
Towards Automated Psychotherapy via Language Modeling0
COVID-19 sentiment analysis via deep learning during the rise of novel cases0
Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion0
Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding0
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network0
IndT5: A Text-to-Text Transformer for 10 Indigenous Languages0
TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling0
ReCAM@IITK at SemEval-2021 Task 4: BERT and ALBERT based Ensemble for Abstract Word PredictionCode0
On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASRCode0
MMBERT: Multimodal BERT Pretraining for Improved Medical VQACode1
ONE: Toward ONE model, ONE algorithm, ONE corpus dedicated to sentiment analysis of Arabic/Arabizi and its dialects0
Emotional RobBERT and Insensitive BERTje: Combining Transformers and Affect Lexica for Dutch Emotion Detection0
Arabic Compact Language Modelling for Resource Limited Devices0
Show:102550
← PrevPage 272 of 353Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Decay RNNValidation perplexity76.67Unverified
2GRUValidation perplexity53.78Unverified
3LSTMValidation perplexity52.73Unverified
4LSTMTest perplexity48.7Unverified
5Temporal CNNTest perplexity45.2Unverified
6TCNTest perplexity45.19Unverified
7GCNN-8Test perplexity44.9Unverified
8Neural cache model (size = 100)Test perplexity44.8Unverified
9Neural cache model (size = 2,000)Test perplexity40.8Unverified
10GPT-2 SmallTest perplexity37.5Unverified
#ModelMetricClaimedVerifiedStatus
1TCNTest perplexity108.47Unverified
2Seq-U-NetTest perplexity107.95Unverified
3GRU (Bai et al., 2018)Test perplexity92.48Unverified
4R-TransformerTest perplexity84.38Unverified
5Zaremba et al. (2014) - LSTM (medium)Test perplexity82.7Unverified
6Gal & Ghahramani (2016) - Variational LSTM (medium)Test perplexity79.7Unverified
7LSTM (Bai et al., 2018)Test perplexity78.93Unverified
8Zaremba et al. (2014) - LSTM (large)Test perplexity78.4Unverified
9Gal & Ghahramani (2016) - Variational LSTM (large)Test perplexity75.2Unverified
10Inan et al. (2016) - Variational RHNTest perplexity66Unverified
#ModelMetricClaimedVerifiedStatus
1LSTM (7 layers)Bit per Character (BPC)1.67Unverified
2HypernetworksBit per Character (BPC)1.34Unverified
3SHA-LSTM (4 layers, h=1024, no attention head)Bit per Character (BPC)1.33Unverified
4LN HM-LSTMBit per Character (BPC)1.32Unverified
5ByteNetBit per Character (BPC)1.31Unverified
6Recurrent Highway NetworksBit per Character (BPC)1.27Unverified
7Large FS-LSTM-4Bit per Character (BPC)1.25Unverified
8Large mLSTMBit per Character (BPC)1.24Unverified
9AWD-LSTM (3 layers)Bit per Character (BPC)1.23Unverified
10Cluster-Former (#C=512)Bit per Character (BPC)1.22Unverified
#ModelMetricClaimedVerifiedStatus
1Smaller Transformer 126M (pre-trained)Test perplexity33Unverified
2OPT 125MTest perplexity32.26Unverified
3Larger Transformer 771M (pre-trained)Test perplexity28.1Unverified
4OPT 1.3BTest perplexity19.55Unverified
5GPT-Neo 125MTest perplexity17.83Unverified
6OPT 2.7BTest perplexity17.81Unverified
7Smaller Transformer 126M (fine-tuned)Test perplexity12Unverified
8GPT-Neo 1.3BTest perplexity11.46Unverified
9Transformer 125MTest perplexity10.7Unverified
10GPT-Neo 2.7BTest perplexity10.44Unverified