Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 4126–4150 of 4240 papers

Title	Date	Tasks	Status
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning	Apr 3, 2019	Incremental LearningKnowledge Distillation	—Unverified
Correlation Congruence for Knowledge Distillation	Apr 3, 2019	Face Recognitionimage-classification	CodeCode Available
A Comprehensive Overhaul of Feature Distillation	Apr 3, 2019	General Classificationimage-classification	CodeCode Available
Making Neural Machine Reading Comprehension Faster	Mar 29, 2019	Knowledge DistillationMachine Reading Comprehension	—Unverified
Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation	Mar 27, 2019	Knowledge DistillationManagement	—Unverified
Improving Neural Architecture Search Image Classifiers via Ensemble Learning	Mar 14, 2019	Ensemble LearningImage Classification	CodeCode Available
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness	Mar 14, 2019	Knowledge Distillation	—Unverified
Knowledge Adaptation for Efficient Semantic Segmentation	Mar 12, 2019	Knowledge DistillationSegmentation	—Unverified
Structured Knowledge Distillation for Dense Prediction	Mar 11, 2019	Depth EstimationGeneral Classification	CodeCode Available
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation	Mar 11, 2019	Depth EstimationDepth Prediction	—Unverified
SeizureNet: Multi-Spectral Deep Feature Learning for Seizure Type Classification	Mar 8, 2019	ClassificationEEG	CodeCode Available
TKD: Temporal Knowledge Distillation for Active Perception	Mar 4, 2019	Knowledge DistillationObject	—Unverified
Multilingual Neural Machine Translation with Knowledge Distillation	Feb 27, 2019	DiversityKnowledge Distillation	CodeCode Available
Improved Knowledge Distillation via Teacher Assistant	Feb 9, 2019	Knowledge Distillation	CodeCode Available
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression	Feb 3, 2019	Knowledge DistillationModel Compression	—Unverified
Compressing GANs using Knowledge Distillation	Feb 1, 2019	Knowledge DistillationSuper-Resolution	—Unverified
Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks	Jan 26, 2019	Knowledge Distillationspeech-recognition	—Unverified
Unsupervised Learning of Neural Networks to Explain Neural Networks (extended abstract)	Jan 21, 2019	Knowledge DistillationObject	—Unverified
Learning Efficient Detector with Semi-supervised Adaptive Distillation	Jan 2, 2019	image-classificationImage Classification	CodeCode Available
Stealing Neural Networks via Timing Side Channels	Dec 31, 2018	Knowledge DistillationReinforcement Learning	—Unverified
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation	Dec 28, 2018	EthicsKnowledge Distillation	—Unverified
Learning Student Networks via Feature Embedding	Dec 17, 2018	Knowledge Distillation	—Unverified
Spatial Knowledge Distillation to aid Visual Reasoning	Dec 10, 2018	DiagnosticKnowledge Distillation	—Unverified
Optimizing speed/accuracy trade-off for person re-identification via knowledge distillation	Dec 7, 2018	Deep LearningGeneral Classification	—Unverified
An Embarrassingly Simple Approach for Knowledge Distillation	Dec 5, 2018	Face RecognitionKnowledge Distillation	CodeCode Available

Show:10 25 50

← PrevPage 166 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified