Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 751–775 of 4240 papers

Title	Date	Tasks	Status	Hype
Transformer-Based Source-Free Domain Adaptation	May 28, 2021	Domain AdaptationKnowledge Distillation	CodeCode Available	1
Selective Knowledge Distillation for Neural Machine Translation	May 27, 2021	Knowledge DistillationMachine Translation	CodeCode Available	1
Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs	May 25, 2021	AttributeKnowledge Distillation	CodeCode Available	1
Backdoor Attacks on Self-Supervised Learning	May 21, 2021	Backdoor AttackInductive Bias	CodeCode Available	1
Data-Free Knowledge Distillation for Heterogeneous Federated Learning	May 20, 2021	Data-free Knowledge DistillationFederated Learning	CodeCode Available	1
Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking	May 20, 2021	Document RankingKnowledge Distillation	CodeCode Available	1
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation	May 19, 2021	Image ClassificationKnowledge Distillation	CodeCode Available	1
Contrastive Model Inversion for Data-Free Knowledge Distillation	May 18, 2021	Contrastive LearningData-free Knowledge Distillation	CodeCode Available	1
Graph-Free Knowledge Distillation for Graph Neural Networks	May 16, 2021	Knowledge DistillationTransfer Learning	CodeCode Available	1
Undistillable: Making A Nasty Teacher That CANNOT teach students	May 16, 2021	Knowledge Distillation	CodeCode Available	1
AgeFlow: Conditional Age Progression and Regression with Normalizing Flows	May 15, 2021	AttributeKnowledge Distillation	CodeCode Available	1
Boosting Light-Weight Depth Estimation Via Knowledge Distillation	May 13, 2021	Computational EfficiencyDepth Estimation	CodeCode Available	1
When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks	May 13, 2021	Knowledge DistillationPose Estimation	CodeCode Available	1
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation	May 12, 2021	Adversarial TextData Augmentation	CodeCode Available	1
Initialization and Regularization of Factorized Neural Layers	May 3, 2021	Knowledge DistillationModel Compression	CodeCode Available	1
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation	Apr 28, 2021	image-classificationImage Classification	CodeCode Available	1
Distilling Audio-Visual Knowledge by Compositional Contrastive Learning	Apr 22, 2021	Audio Taggingaudio-visual learning	CodeCode Available	1
Balanced Knowledge Distillation for Long-tailed Learning	Apr 21, 2021	Knowledge Distillation	CodeCode Available	1
Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices	Apr 21, 2021	Face GenerationFace Model	CodeCode Available	1
Distill on the Go: Online knowledge distillation in self-supervised learning	Apr 20, 2021	Knowledge DistillationSelf-Supervised Learning	CodeCode Available	1
Distilling Knowledge via Knowledge Review	Apr 19, 2021	Instance SegmentationKnowledge Distillation	CodeCode Available	1
On Learning the Geodesic Path for Incremental Learning	Apr 17, 2021	Incremental LearningKnowledge Distillation	CodeCode Available	1
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos	Apr 16, 2021	Activity RecognitionDiversity	CodeCode Available	1
Counter-Interference Adapter for Multilingual Machine Translation	Apr 16, 2021	Knowledge DistillationMachine Translation	CodeCode Available	1
Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer	Apr 13, 2021	Domain AdaptationIncremental Learning	CodeCode Available	1

Show:10 25 50

← PrevPage 31 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified