Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2026–2050 of 4240 papers

Title	Date	Tasks	Status	Hype
Self-Distillation for Gaussian Process Regression and Classification	Apr 5, 2023	ClassificationGPR	CodeCode Available	0
MadEye: Boosting Live Video Analytics Accuracy with Adaptive Camera Configurations	Apr 4, 2023	Knowledge Distillation	—Unverified	0
Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher	Apr 4, 2023	Federated LearningKnowledge Distillation	CodeCode Available	1
Cross-Class Feature Augmentation for Class Incremental Learning	Apr 4, 2023	class-incremental learningClass Incremental Learning	—Unverified	0
Knowledge-Distilled Graph Neural Networks for Personalized Epileptic Seizure Detection	Apr 3, 2023	channel selectionEEG	—Unverified	0
Vision-Language Models for Vision Tasks: A Survey	Apr 3, 2023	BenchmarkingKnowledge Distillation	CodeCode Available	4
Domain Generalization for Crop Segmentation with Standardized Ensemble Knowledge Distillation	Apr 3, 2023	Domain GeneralizationKnowledge Distillation	CodeCode Available	0
A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation	Apr 2, 2023	Face GenerationKnowledge Distillation	—Unverified	0
Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation	Mar 31, 2023	Knowledge DistillationMachine Translation	—Unverified	0
Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders	Mar 31, 2023	Knowledge DistillationLanguage Modeling	—Unverified	0
GVP: Generative Volumetric Primitives	Mar 31, 2023	Image GenerationKnowledge Distillation	—Unverified	0
Knowledge Distillation for Feature Extraction in Underwater VSLAM	Mar 31, 2023	BinarizationKnowledge Distillation	CodeCode Available	1
oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes	Mar 30, 2023	Knowledge DistillationModel Compression	—Unverified	0
If At First You Don't Succeed: Test Time Re-ranking for Zero-shot, Cross-domain Retrieval	Mar 30, 2023	Image RetrievalKnowledge Distillation	—Unverified	0
Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuning	Mar 30, 2023	Continual LearningKnowledge Distillation	CodeCode Available	1
KD-DLGAN: Data Limited Image Generation via Knowledge Distillation	Mar 30, 2023	DiversityImage Generation	—Unverified	0
Asymmetric Image Retrieval with Cross Model Compatible Ensembles	Mar 30, 2023	DiversityFace Recognition	—Unverified	0
SimDistill: Simulated Multi-modal Distillation for BEV 3D Object Detection	Mar 29, 2023	3D geometry3D Object Detection	CodeCode Available	1
Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels	Mar 28, 2023	Knowledge Distillation	CodeCode Available	1
Information-Theoretic GAN Compression with Variational Energy-based Model	Mar 28, 2023	Image EnhancementKnowledge Distillation	—Unverified	0
HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models	Mar 28, 2023	DecoderHuman-Object Interaction Detection	CodeCode Available	1
SELF-VS: Self-supervised Encoding Learning For Video Summarization	Mar 28, 2023	Knowledge DistillationRepresentation Learning	CodeCode Available	0
Projected Latent Distillation for Data-Agnostic Consolidation in Distributed Continual Learning	Mar 28, 2023	Continual LearningKnowledge Distillation	CodeCode Available	0
DisWOT: Student Architecture Search for Distillation WithOut Training	Mar 28, 2023	Knowledge Distillation	CodeCode Available	1
Improving Neural Topic Models with Wasserstein Knowledge Distillation	Mar 27, 2023	Knowledge DistillationTopic Models	CodeCode Available	0

Show:10 25 50

← PrevPage 82 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified