Language Modelling

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 10976–11000 of 17610 papers

Title	Date	Tasks	Status	Hype
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers	Oct 31, 2022	GPULanguage Modelling	CodeCode Available	7
Emergent Linguistic Structures in Neural Networks are Fragile	Oct 31, 2022	Language Modelling	CodeCode Available	0
CodeEditor: Learning to Edit Source Code with Pre-trained Models	Oct 31, 2022	Language ModelingLanguage Modelling	CodeCode Available	0
Artificial intelligence in government: Concepts, standards, and a unified framework	Oct 31, 2022	Language Modelling	CodeCode Available	0
Tables to LaTeX: structure and content extraction from scientific tables	Oct 31, 2022	Language ModelingLanguage Modelling	—Unverified	0
Modular Hybrid Autoregressive Transducer	Oct 31, 2022	DecoderLanguage Modeling	—Unverified	0
L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning	Oct 31, 2022	image-classificationImage Classification	CodeCode Available	1
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control	Oct 31, 2022	DiversityLanguage Modeling	CodeCode Available	1
token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text	Oct 30, 2022	intent-classificationIntent Classification	—Unverified	0
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts	Oct 30, 2022	Language ModelingLanguage Modelling	—Unverified	0
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model	Oct 29, 2022	Language ModelingLanguage Modelling	—Unverified	0
Exploiting prompt learning with pre-trained language models for Alzheimer's Disease detection	Oct 29, 2022	Alzheimer's Disease DetectionLanguage Modelling	CodeCode Available	0
Differentiable Data Augmentation for Contrastive Sentence Representation Learning	Oct 29, 2022	Contrastive LearningData Augmentation	CodeCode Available	1
NTULM: Enriching Social Media Text Representations with Non-Textual Units	Oct 29, 2022	Language ModelingLanguage Modelling	—Unverified	0
Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models	Oct 28, 2022	Common Sense ReasoningCoreference Resolution	—Unverified	0
Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion	Oct 28, 2022	Emotion RecognitionLanguage Modeling	CodeCode Available	1
Modeling structure-building in the brain with CCG parsing and large language models	Oct 28, 2022	Language Modelling	—Unverified	0
RoChBert: Towards Robust BERT Fine-tuning for Chinese	Oct 28, 2022	Data AugmentationLanguage Modeling	CodeCode Available	1
You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM	Oct 28, 2022	Language ModelingLanguage Modelling	CodeCode Available	0
UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance	Oct 28, 2022	Image GenerationImage-text matching	—Unverified	0
Feature Engineering vs BERT on Twitter Data	Oct 28, 2022	Feature EngineeringGPU	—Unverified	0
DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention	Oct 28, 2022	Image CaptioningLanguage Modeling	—Unverified	0
Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training	Oct 28, 2022	Language ModellingMotion Generation	CodeCode Available	1
Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge	Oct 27, 2022	Acoustic Unit DiscoveryLanguage Modeling	—Unverified	0
Nearest Neighbor Language Models for Stylistic Controllable Generation	Oct 27, 2022	Language ModelingLanguage Modelling	—Unverified	0

Show:10 25 50

← PrevPage 440 of 705Next →

All datasets WikiText-103 Penn Treebank (Word Level)enwik8 The Pile WikiText-2 LAMBADA One Billion Word Text8 Penn Treebank (Character Level)Hutter Prize OpenWebText SALMon

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Decay RNN	Validation perplexity	76.67	—	Unverified
2	GRU	Validation perplexity	53.78	—	Unverified
3	LSTM	Validation perplexity	52.73	—	Unverified
4	LSTM	Test perplexity	48.7	—	Unverified
5	Temporal CNN	Test perplexity	45.2	—	Unverified
6	TCN	Test perplexity	45.19	—	Unverified
7	GCNN-8	Test perplexity	44.9	—	Unverified
8	Neural cache model (size = 100)	Test perplexity	44.8	—	Unverified
9	Neural cache model (size = 2,000)	Test perplexity	40.8	—	Unverified
10	GPT-2 Small	Test perplexity	37.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TCN	Test perplexity	108.47	—	Unverified
2	Seq-U-Net	Test perplexity	107.95	—	Unverified
3	GRU (Bai et al., 2018)	Test perplexity	92.48	—	Unverified
4	R-Transformer	Test perplexity	84.38	—	Unverified
5	Zaremba et al. (2014) - LSTM (medium)	Test perplexity	82.7	—	Unverified
6	Gal & Ghahramani (2016) - Variational LSTM (medium)	Test perplexity	79.7	—	Unverified
7	LSTM (Bai et al., 2018)	Test perplexity	78.93	—	Unverified
8	Zaremba et al. (2014) - LSTM (large)	Test perplexity	78.4	—	Unverified
9	Gal & Ghahramani (2016) - Variational LSTM (large)	Test perplexity	75.2	—	Unverified
10	Inan et al. (2016) - Variational RHN	Test perplexity	66	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSTM (7 layers)	Bit per Character (BPC)	1.67	—	Unverified
2	Hypernetworks	Bit per Character (BPC)	1.34	—	Unverified
3	SHA-LSTM (4 layers, h=1024, no attention head)	Bit per Character (BPC)	1.33	—	Unverified
4	LN HM-LSTM	Bit per Character (BPC)	1.32	—	Unverified
5	ByteNet	Bit per Character (BPC)	1.31	—	Unverified
6	Recurrent Highway Networks	Bit per Character (BPC)	1.27	—	Unverified
7	Large FS-LSTM-4	Bit per Character (BPC)	1.25	—	Unverified
8	Large mLSTM	Bit per Character (BPC)	1.24	—	Unverified
9	AWD-LSTM (3 layers)	Bit per Character (BPC)	1.23	—	Unverified
10	Cluster-Former (#C=512)	Bit per Character (BPC)	1.22	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Smaller Transformer 126M (pre-trained)	Test perplexity	33	—	Unverified
2	OPT 125M	Test perplexity	32.26	—	Unverified
3	Larger Transformer 771M (pre-trained)	Test perplexity	28.1	—	Unverified
4	OPT 1.3B	Test perplexity	19.55	—	Unverified
5	GPT-Neo 125M	Test perplexity	17.83	—	Unverified
6	OPT 2.7B	Test perplexity	17.81	—	Unverified
7	Smaller Transformer 126M (fine-tuned)	Test perplexity	12	—	Unverified
8	GPT-Neo 1.3B	Test perplexity	11.46	—	Unverified
9	Transformer 125M	Test perplexity	10.7	—	Unverified
10	GPT-Neo 2.7B	Test perplexity	10.44	—	Unverified