Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3776–3800 of 4240 papers

Title	Date	Tasks	Status	Hype
Teacher-Student Training and Triplet Loss for Facial Expression Recognition under Occlusion	Aug 3, 2020	Facial Expression RecognitionFacial Expression Recognition (FER)	—Unverified	0
Differentiable Feature Aggregation Search for Knowledge Distillation	Aug 2, 2020	Knowledge DistillationModel Compression	—Unverified	0
Weight Decay Scheduling and Knowledge Distillation for Active Learning	Aug 1, 2020	Active LearningKnowledge Distillation	—Unverified	0
Improving Knowledge Distillation via Category Structure	Aug 1, 2020	Knowledge Distillation	CodeCode Available	1
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition	Aug 1, 2020	DiversityFace Recognition	—Unverified	0
Feature Normalized Knowledge Distillation for Image Classification	Aug 1, 2020	ClassificationGeneral Classification	CodeCode Available	0
Local Correlation Consistency for Knowledge Distillation	Aug 1, 2020	Knowledge Distillation	—Unverified	0
Intra-class Feature Variation Distillation for Semantic Segmentation	Aug 1, 2020	Knowledge DistillationSegmentation	CodeCode Available	1
YOLO in the Dark - Domain Adaptation Method for Merging Multiple Models -	Aug 1, 2020	Domain AdaptationKnowledge Distillation	—Unverified	0
AMLN: Adversarial-based Mutual Learning Network for Online Knowledge Distillation	Aug 1, 2020	Knowledge DistillationTransfer Learning	—Unverified	0
Distilling Visual Priors from Self-Supervised Learning	Aug 1, 2020	ClassificationContrastive Learning	CodeCode Available	1
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge	Jul 28, 2020	Federated LearningKnowledge Distillation	CodeCode Available	1
Weakly Supervised 3D Object Detection from Point Clouds	Jul 28, 2020	3D Object DetectionKnowledge Distillation	CodeCode Available	1
Dynamic Knowledge Distillation for Black-box Hypothesis Transfer Learning	Jul 24, 2020	Knowledge DistillationTransfer Learning	—Unverified	0
Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation	Jul 21, 2020	Instance SegmentationKnowledge Distillation	CodeCode Available	1
Interpretable Foreground Object Search As Knowledge Distillation	Jul 20, 2020	Knowledge DistillationObject	—Unverified	0
Multi-label Contrastive Predictive Coding	Jul 20, 2020	Knowledge DistillationMulti-class Classification	—Unverified	0
Resolution Switchable Networks for Runtime Efficient Image Recognition	Jul 19, 2020	Knowledge DistillationQuantization	CodeCode Available	1
Knowledge Distillation in Deep Learning and its Applications	Jul 17, 2020	Deep LearningKnowledge Distillation	—Unverified	0
CovidCare: Transferring Knowledge from Existing EMR to Emerging Epidemic for Interpretable Prognosis	Jul 17, 2020	DiagnosticKnowledge Distillation	—Unverified	0
Defocus Blur Detection via Depth Distillation	Jul 16, 2020	DecoderDefocus Blur Detection	CodeCode Available	1
Self-supervision on Unlabelled OR Data for Multi-person 2D/3D Human Pose Estimation	Jul 16, 2020	3D Human Pose Estimation3D Pose Estimation	CodeCode Available	1
UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data	Jul 15, 2020	Cross-Lingual NERCross-Lingual Transfer	CodeCode Available	0
Add a SideNet to your MainNet	Jul 14, 2020	General ClassificationKnowledge Distillation	—Unverified	0
P-KDGAN: Progressive Knowledge Distillation with GANs for One-class Novelty Detection	Jul 14, 2020	Anomaly DetectionDecoder	—Unverified	0

Show:10 25 50

← PrevPage 152 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified