Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2776–2800 of 4240 papers

Title	Date	Tasks	Status	Hype
Learning to Purification for Unsupervised Person Re-identification	Apr 21, 2022	Knowledge DistillationPerson Re-Identification	—Unverified	0
Eliminating Backdoor Triggers for Deep Neural Networks Using Attention Relation Graph Distillation	Apr 21, 2022	backdoor defenseKnowledge Distillation	CodeCode Available	1
Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation	Apr 21, 2022	Instance SegmentationKnowledge Distillation	—Unverified	0
HRPose: Real-Time High-Resolution 6D Pose Estimation Network Using Knowledge Distillation	Apr 20, 2022	6D Pose Estimation6D Pose Estimation using RGB	—Unverified	0
DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue Generation	Apr 19, 2022	Dialogue GenerationKnowledge Distillation	CodeCode Available	1
Modeling Missing Annotations for Incremental Learning in Object Detection	Apr 19, 2022	Incremental LearningInstance Segmentation	CodeCode Available	1
Multi-Modal Few-Shot Object Detection with Meta-Learning-Based Cross-Modal Prompting	Apr 16, 2022	Few-Shot LearningFew-Shot Object Detection	—Unverified	0
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation	Apr 15, 2022	Knowledge DistillationMixture-of-Experts	CodeCode Available	1
CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation	Apr 15, 2022	Contrastive LearningData Augmentation	—Unverified	0
Ensemble diverse hypotheses and knowledge distillation for unsupervised cross-subject adaptation	Apr 15, 2022	Activity RecognitionDomain Adaptation	CodeCode Available	0
Cross-Image Relational Knowledge Distillation for Semantic Segmentation	Apr 14, 2022	Knowledge DistillationSegmentation	CodeCode Available	2
Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection	Apr 14, 2022	Knowledge DistillationMultiple Instance Learning	—Unverified	0
Impossible Triangle: What's Next for Pre-trained Language Models?	Apr 13, 2022	Data AugmentationFew-Shot Learning	—Unverified	0
Localization Distillation for Object Detection	Apr 12, 2022	Knowledge DistillationObject	CodeCode Available	2
DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimization	Apr 12, 2022	Knowledge DistillationMeta-Learning	—Unverified	0
CoupleFace: Relation Matters for Face Recognition Distillation	Apr 12, 2022	Face RecognitionKnowledge Distillation	—Unverified	0
LRH-Net: A Multi-Level Knowledge Distillation Approach for Low-Resource Heart Network	Apr 11, 2022	Knowledge Distillation	CodeCode Available	1
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results	Apr 7, 2022	Image ClassificationKnowledge Distillation	CodeCode Available	2
Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation	Apr 5, 2022	Class-Incremental Object DetectionIncremental Learning	CodeCode Available	1
Towards On-Board Panoptic Segmentation of Multispectral Satellite Images	Apr 5, 2022	Knowledge DistillationPanoptic Segmentation	—Unverified	0
Using Explainable Boosting Machine to Compare Idiographic and Nomothetic Approaches for Ecological Momentary Assessment Data	Apr 4, 2022	Interpretable Machine LearningKnowledge Distillation	—Unverified	0
Co-Teaching for Unsupervised Domain Adaptation and Expansion	Apr 4, 2022	Domain Adaptationimage-classification	CodeCode Available	0
CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning	Apr 4, 2022	Federated LearningKnowledge Distillation	—Unverified	0
DST: Dynamic Substitute Training for Data-free Black-box Attack	Apr 3, 2022	Knowledge Distillation	—Unverified	0
CL-XABSA: Contrastive Learning for Cross-lingual Aspect-based Sentiment Analysis	Apr 2, 2022	Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)	CodeCode Available	0

Show:10 25 50

← PrevPage 112 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified