Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1626–1650 of 4240 papers

Title	Date	Tasks	Status	Score
Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding	Aug 14, 2019	Knowledge DistillationNatural Language Understanding	CodeCode Available	5
Gradient Knowledge Distillation for Pre-trained Language Models	Nov 2, 2022	Knowledge Distillation	CodeCode Available	5
Distilling Knowledge for Empathy Detection	Nov 1, 2021	Knowledge Distillation	CodeCode Available	5
Distilling Knowledge for Designing Computational Imaging Systems	Jan 29, 2025	DecoderImage Reconstruction	CodeCode Available	5
GOTHAM: Graph Class Incremental Learning Framework under Weak Supervision	Apr 7, 2025	Attributeclass-incremental learning	CodeCode Available	5
Spending Your Winning Lottery Better After Drawing It	Jan 8, 2021	Knowledge Distillation	CodeCode Available	5
Goldfish: An Efficient Federated Unlearning Framework	Apr 4, 2024	Knowledge DistillationMachine Unlearning	CodeCode Available	5
Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks	Oct 2, 2023	Knowledge DistillationNode Classification	CodeCode Available	5
Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?	Jul 31, 2023	Contrastive LearningKnowledge Distillation	CodeCode Available	5
Goal-Conditioned Q-Learning as Knowledge Distillation	Aug 28, 2022	Knowledge DistillationQ-Learning	CodeCode Available	5
Few-shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge Distillation	Aug 5, 2023	Class-Incremental Semantic SegmentationKnowledge Distillation	CodeCode Available	5
Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation	May 16, 2024	Dialogue GenerationKnowledge Distillation	CodeCode Available	5
Distilling Image Dehazing With Heterogeneous Task Imitation	Jun 1, 2020	image-classificationImage Classification	CodeCode Available	5
GNN's Uncertainty Quantification using Self-Distillation	Jun 24, 2025	Knowledge DistillationUncertainty Quantification	CodeCode Available	5
GLANCE: Global to Local Architecture-Neutral Concept-based Explanations	Jul 5, 2022	DisentanglementFeature Importance	CodeCode Available	5
GLiRA: Black-Box Membership Inference Attack via Knowledge Distillation	May 13, 2024	image-classificationImage Classification	CodeCode Available	5
Distilling Global and Local Logits With Densely Connected Relations	Jan 1, 2021	image-classificationImage Classification	CodeCode Available	5
GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference	Apr 8, 2021	Disease Predictiongraph construction	CodeCode Available	5
Distilling Focal Knowledge From Imperfect Expert for 3D Object Detection	Jan 1, 2023	3D geometry3D Object Detection	CodeCode Available	5
GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment	May 30, 2024	GSM8KKnowledge Distillation	CodeCode Available	5
Camera-Incremental Object Re-Identification with Identity Knowledge Evolution	May 25, 2023	Knowledge DistillationObject	CodeCode Available	5
Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Prediction	Jan 16, 2024	Instance SegmentationKnowledge Distillation	CodeCode Available	5
Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing	May 31, 2021	Knowledge DistillationUnsupervised Pre-training	CodeCode Available	5
Annealing Knowledge Distillation	Apr 14, 2021	image-classificationImage Classification	CodeCode Available	5
Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression	Apr 7, 2021	General Classificationimage-classification	CodeCode Available	5

Show:10 25 50

← PrevPage 66 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified