Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3876–3900 of 4240 papers

Title	Date	Tasks	Status	Hype
Language Model Prior for Low-Resource Neural Machine Translation	Apr 30, 2020	Knowledge DistillationLanguage Modeling	CodeCode Available	1
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference	Apr 29, 2020	Knowledge DistillationQuantization	—Unverified	0
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning	Apr 27, 2020	Knowledge DistillationLanguage Modeling	—Unverified	0
A Tailored Pre-Training Model for Task-Oriented Dialog Generation	Apr 24, 2020	Knowledge DistillationLanguage Modeling	CodeCode Available	0
Distilling Knowledge from Refinement in Multiple Instance Detection Networks	Apr 23, 2020	Knowledge DistillationMultiple Instance Learning	CodeCode Available	1
A Study of Non-autoregressive Model for Sequence Generation	Apr 22, 2020	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation	Apr 21, 2020	Knowledge DistillationSentence	CodeCode Available	1
Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation	Apr 21, 2020	DecoderKnowledge Distillation	—Unverified	0
Role-Wise Data Augmentation for Knowledge Distillation	Apr 19, 2020	Data AugmentationKnowledge Distillation	CodeCode Available	1
Triplet Loss for Knowledge Distillation	Apr 17, 2020	Knowledge DistillationMetric Learning	CodeCode Available	1
Multimodal and multiview distillation for real-time player detection on a football field	Apr 16, 2020	Data AugmentationKnowledge Distillation	CodeCode Available	1
Knowledge Distillation for Action Anticipation via Label Smoothing	Apr 16, 2020	Action AnticipationAutonomous Driving	—Unverified	0
Dark Experience for General Continual Learning: a Strong, Simple Baseline	Apr 15, 2020	class-incremental learningClass Incremental Learning	CodeCode Available	1
Building a Multi-domain Neural Machine Translation Model using Knowledge Distillation	Apr 15, 2020	Domain AdaptationKnowledge Distillation	—Unverified	0
Smart Inference for Multidigit Convolutional Neural Network based Barcode Decoding	Apr 14, 2020	Knowledge Distillation	—Unverified	0
Towards Robust Classification with Image Quality Assessment	Apr 14, 2020	ClassificationGeneral Classification	—Unverified	0
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks	Apr 13, 2020	Knowledge DistillationModel Compression	CodeCode Available	2
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models	Apr 12, 2020	Knowledge Distillationnamed-entity-recognition	—Unverified	0
KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflow	Apr 11, 2020	CPUGPU	CodeCode Available	1
Inter-Region Affinity Distillation for Road Marking Segmentation	Apr 11, 2020	Knowledge DistillationLane Detection	CodeCode Available	1
Knowledge Distillation for Mobile Edge Computation Offloading	Apr 9, 2020	Imitation LearningKnowledge Distillation	—Unverified	0
On the Effect of Dropping Layers of Pre-trained Transformer Models	Apr 8, 2020	Knowledge DistillationSentence	CodeCode Available	1
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression	Apr 8, 2020	BlockingKnowledge Distillation	—Unverified	0
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling	Apr 8, 2020	Aspect ExtractionKnowledge Distillation	CodeCode Available	1
Towards Efficient Unconstrained Palmprint Recognition via Deep Distillation Hashing	Apr 7, 2020	Knowledge Distillation	CodeCode Available	1

Show:10 25 50

← PrevPage 156 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified