Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3676–3700 of 4240 papers

Title	Date	Tasks	Status
Tiny models from tiny data: Textual and null-text inversion for few-shot distillation	Jun 5, 2024	Few-Shot Image Classificationimage-classification	CodeCode Available
Intra-class Patch Swap for Self-Distillation	May 20, 2025	image-classificationImage Classification	CodeCode Available
Distill n' Explain: explaining graph neural networks using simple surrogates	Mar 17, 2023	Knowledge Distillation	CodeCode Available
Interpreting Microbiome Relative Abundance Data Using Symbolic Regression	Oct 18, 2024	DiagnosticKnowledge Distillation	CodeCode Available
Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidance	Dec 7, 2023	Domain AdaptationKnowledge Distillation	CodeCode Available
RDPD: Rich Data Helps Poor Data via Imitation	Sep 6, 2018	Knowledge Distillation	CodeCode Available
Interpreting and Disentangling Feature Components of Various Complexity from DNNs	Jun 29, 2020	Knowledge Distillation	CodeCode Available
Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network	Apr 28, 2021	Graph Neural NetworkKnowledge Distillation	CodeCode Available
BAM! Born-Again Multi-Task Networks for Natural Language Understanding	Jul 10, 2019	Knowledge DistillationNatural Language Understanding	CodeCode Available
m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers	Feb 26, 2024	Knowledge DistillationMixture-of-Experts	CodeCode Available
3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding	Feb 28, 2024	document understandingForm	CodeCode Available
Distilling Virtual Examples for Long-tailed Recognition	Mar 28, 2021	Knowledge DistillationLong-tail Learning	CodeCode Available
Inter-Domain Alignment for Predicting High-Resolution Brain Networks Using Teacher-Student Learning	Oct 6, 2021	DecoderDomain Adaptation	CodeCode Available
Instance Temperature Knowledge Distillation	Jun 27, 2024	Decision MakingEfficient Exploration	CodeCode Available
Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation	Jul 18, 2024	Knowledge DistillationSemantic Segmentation	CodeCode Available
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism	Oct 19, 2020	DecoderKnowledge Distillation	CodeCode Available
Real-Time Cell Sorting with Scalable In Situ FPGA-Accelerated Deep Learning	Mar 16, 2025	Cell DetectionClassification	CodeCode Available
Real-Time Correlation Tracking via Joint Model Compression and Transfer	Jul 23, 2019	Computational EfficiencyCPU	CodeCode Available
Real-Time Decentralized knowledge Transfer at the Edge	Nov 11, 2020	Knowledge DistillationTransfer Learning	CodeCode Available
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data	Jul 7, 2023	Knowledge DistillationModel Compression	CodeCode Available
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations	Sep 13, 2018	Depth EstimationGPU	CodeCode Available
Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial Bias	Mar 1, 2023	Backdoor AttackKnowledge Distillation	CodeCode Available
AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning	Jan 1, 2025	Audio-visual Question AnsweringContinual Learning	CodeCode Available
Aligning Logits Generatively for Principled Black-Box Knowledge Distillation	May 21, 2022	Federated LearningKnowledge Distillation	CodeCode Available
Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation	Jun 10, 2021	Knowledge Distillation	CodeCode Available

Show:10 25 50

← PrevPage 148 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified