Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 4126–4150 of 4240 papers

Title	Date	Tasks	Status
Improved training of binary networks for human pose estimation and image recognition	Apr 11, 2019	BinarizationClassification with Binary Neural Network	—Unverified
Variational Information Distillation for Knowledge Transfer	Apr 11, 2019	Knowledge DistillationTransfer Learning	—Unverified
Knowledge Squeezed Adversarial Network Compression	Apr 10, 2019	Knowledge DistillationTransfer Learning	—Unverified
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency	Apr 10, 2019	GPUKnowledge Distillation	—Unverified
Relational Knowledge Distillation	Apr 10, 2019	Knowledge DistillationMetric Learning	CodeCode Available
Ultrafast Video Attention Prediction with Coupled Knowledge Distillation	Apr 9, 2019	CPUGPU	—Unverified
Knowledge Distillation for Human Action Anticipation	Apr 9, 2019	Action AnticipationAction Recognition	—Unverified
Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization	Apr 8, 2019	Knowledge DistillationLanguage Modeling	—Unverified
Long-Term Vehicle Localization by Recursive Knowledge Distillation	Apr 7, 2019	Domain AdaptationEnsemble Learning	—Unverified
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion	Apr 6, 2019	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning	Apr 3, 2019	Incremental LearningKnowledge Distillation	—Unverified
Correlation Congruence for Knowledge Distillation	Apr 3, 2019	Face Recognitionimage-classification	CodeCode Available
A Comprehensive Overhaul of Feature Distillation	Apr 3, 2019	General Classificationimage-classification	CodeCode Available
Making Neural Machine Reading Comprehension Faster	Mar 29, 2019	Knowledge DistillationMachine Reading Comprehension	—Unverified
Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation	Mar 27, 2019	Knowledge DistillationManagement	—Unverified
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness	Mar 14, 2019	Knowledge Distillation	—Unverified
Improving Neural Architecture Search Image Classifiers via Ensemble Learning	Mar 14, 2019	Ensemble LearningImage Classification	CodeCode Available
Knowledge Adaptation for Efficient Semantic Segmentation	Mar 12, 2019	Knowledge DistillationSegmentation	—Unverified
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation	Mar 11, 2019	Depth EstimationDepth Prediction	—Unverified
Structured Knowledge Distillation for Dense Prediction	Mar 11, 2019	Depth EstimationGeneral Classification	CodeCode Available
SeizureNet: Multi-Spectral Deep Feature Learning for Seizure Type Classification	Mar 8, 2019	ClassificationEEG	CodeCode Available
TKD: Temporal Knowledge Distillation for Active Perception	Mar 4, 2019	Knowledge DistillationObject	—Unverified
Multilingual Neural Machine Translation with Knowledge Distillation	Feb 27, 2019	DiversityKnowledge Distillation	CodeCode Available
Improved Knowledge Distillation via Teacher Assistant	Feb 9, 2019	Knowledge Distillation	CodeCode Available
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression	Feb 3, 2019	Knowledge DistillationModel Compression	—Unverified

Show:10 25 50

← PrevPage 166 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified