Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1151–1175 of 4240 papers

Title	Date	Tasks	Status	Score
A Knowledge Distillation Ensemble Framework for Predicting Short and Long-term Hospitalisation Outcomes from Electronic Health Records Data	Nov 18, 2020	Decision MakingICU Admission	CodeCode Available	5
Knowledge Distillation from Cross Teaching Teachers for Efficient Semi-Supervised Abdominal Organ Segmentation in CT	Nov 11, 2022	Image SegmentationKnowledge Distillation	CodeCode Available	5
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary	May 15, 2018	Adversarial AttackKnowledge Distillation	CodeCode Available	5
Knowledge Distillation for Multi-Target Domain Adaptation in Real-Time Person Re-Identification	May 12, 2022	Domain AdaptationKnowledge Distillation	CodeCode Available	5
Autoregressive Knowledge Distillation through Imitation Learning	Sep 15, 2020	Imitation LearningKnowledge Distillation	CodeCode Available	5
Correlation Congruence for Knowledge Distillation	Apr 3, 2019	Face Recognitionimage-classification	CodeCode Available	5
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks	Apr 23, 2019	Image GenerationKnowledge Distillation	CodeCode Available	5
Knowledge Distillation for Quality Estimation	Jul 1, 2021	Data AugmentationKnowledge Distillation	CodeCode Available	5
A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models	Feb 21, 2025	Decision MakingKnowledge Distillation	CodeCode Available	5
Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling	Nov 15, 2022	General KnowledgeKnowledge Distillation	CodeCode Available	5
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation	Jul 6, 2021	Continual LearningDomain Adaptation	CodeCode Available	5
Automatic Assignment of Radiology Examination Protocols Using Pre-trained Language Models with Knowledge Distillation	Sep 1, 2020	Data AugmentationKnowledge Distillation	CodeCode Available	5
Knowledge Distillation for End-to-End Person Search	Sep 3, 2019	Knowledge DistillationModel Compression	CodeCode Available	5
Knowledge Distillation for Singing Voice Detection	Nov 9, 2020	Information RetrievalKnowledge Distillation	CodeCode Available	5
Cooperative Retriever and Ranker in Deep Recommenders	Jun 28, 2022	Knowledge DistillationRecommendation Systems	CodeCode Available	5
Automatic adaptation of object detectors to new domains using self-training	Apr 15, 2019	Domain AdaptationKnowledge Distillation	CodeCode Available	5
Cooperative Knowledge Distillation: A Learner Agnostic Approach	Feb 2, 2024	counterfactualKnowledge Distillation	CodeCode Available	5
Automated Knowledge Distillation via Monte Carlo Tree Search	Jan 1, 2023	image-classificationImage Classification	CodeCode Available	5
Knowledge Distillation-Based Model Extraction Attack using GAN-based Private Counterfactual Explanations	Apr 4, 2024	counterfactualKnowledge Distillation	CodeCode Available	5
Knowledge Distillation by On-the-Fly Native Ensemble	Jun 12, 2018	Computational Efficiencyimage-classification	CodeCode Available	5
Cooperative Classification and Rationalization for Graph Generalization	Mar 10, 2024	ClassificationGraph Classification	CodeCode Available	5
Knowledge Distillation as Semiparametric Inference	Apr 20, 2021	Knowledge DistillationModel Compression	CodeCode Available	5
Knowledge Distillation approach towards Melanoma Detection	Oct 14, 2022	Knowledge DistillationTAG	CodeCode Available	5
Knowledge Distillation By Sparse Representation Matching	Mar 31, 2021	Knowledge DistillationRepresentation Learning	CodeCode Available	5
Knowledge Distillation For Wireless Edge Learning	Apr 3, 2021	Cloud ComputingFederated Learning	CodeCode Available	5

Show:10 25 50

← PrevPage 47 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified