Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2501–2525 of 4240 papers

Title	Date	Tasks	Status	Hype
Towards Explaining Autonomy with Verbalised Decision Tree States	Sep 28, 2022	Knowledge Distillation	—Unverified	0
Efficient On-Device Session-Based Recommendation	Sep 27, 2022	Knowledge DistillationModel Compression	CodeCode Available	1
PROD: Progressive Distillation for Dense Retrieval	Sep 27, 2022	Knowledge DistillationNatural Questions	—Unverified	0
Rethinking Resolution in the Context of Efficient Video Recognition	Sep 26, 2022	Knowledge DistillationVideo Recognition	CodeCode Available	1
Knowledge Distillation to Ensemble Global and Interpretable Prototype-Based Mammogram Classification Models	Sep 26, 2022	DiversityKnowledge Distillation	—Unverified	0
Joint Speech Activity and Overlap Detection with Multi-Exit Architecture	Sep 24, 2022	Action DetectionActivity Detection	—Unverified	0
DRKF: Distilled Rotated Kernel Fusion for Efficient Rotation Invariant Descriptors in Local Feature Matching	Sep 22, 2022	Knowledge Distillation	—Unverified	0
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation	Sep 21, 2022	Data-free Knowledge DistillationKnowledge Distillation	—Unverified	0
Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation	Sep 20, 2022	Data AugmentationKnowledge Distillation	CodeCode Available	0
Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition	Sep 17, 2022	Knowledge DistillationMixture-of-Experts	—Unverified	0
Continual Learning for Class- and Domain-Incremental Semantic Segmentation	Sep 16, 2022	class-incremental learningClass Incremental Learning	—Unverified	0
Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labels	Sep 16, 2022	Knowledge DistillationSelf-Supervised Learning	CodeCode Available	1
Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation	Sep 16, 2022	class-incremental learningClass Incremental Learning	—Unverified	0
Deliberated Domain Bridging for Domain Adaptive Semantic Segmentation	Sep 16, 2022	Domain AdaptationImage-to-Image Translation	CodeCode Available	1
CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation	Sep 15, 2022	Knowledge Distillation	—Unverified	0
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation	Sep 15, 2022	Knowledge DistillationRepresentation Learning	—Unverified	0
On-Device Domain Generalization	Sep 15, 2022	Data AugmentationDomain Generalization	CodeCode Available	2
TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation	Sep 14, 2022	Activity RecognitionHuman Activity Recognition	—Unverified	0
PlaStIL: Plastic and Stable Memory-Free Class-Incremental Learning	Sep 14, 2022	class-incremental learningClass Incremental Learning	—Unverified	0
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching	Sep 13, 2022	Contrastive LearningKnowledge Distillation	CodeCode Available	0
Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation	Sep 12, 2022	Continual LearningData Augmentation	—Unverified	0
Switchable Online Knowledge Distillation	Sep 12, 2022	Knowledge Distillation	CodeCode Available	1
Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation	Sep 10, 2022	Federated Learningimage-classification	—Unverified	0
In-situ animal behavior classification using knowledge distillation and fixed-point quantization	Sep 9, 2022	ClassificationKnowledge Distillation	—Unverified	0
Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning	Sep 9, 2022	Continual LearningKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 101 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified