Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3476–3500 of 4240 papers

Title	Date	Tasks	Status	Hype
A Practical Survey on Faster and Lighter Transformers	Mar 26, 2021	Knowledge DistillationSurvey	—Unverified	0
Distilling Object Detectors via Decoupled Features	Mar 26, 2021	image-classificationImage Classification	CodeCode Available	1
Hands-on Guidance for Distilling Object Detectors	Mar 26, 2021	Knowledge DistillationObject	—Unverified	0
Leaning Compact and Representative Features for Cross-Modality Person Re-Identification	Mar 26, 2021	Cross-Modality Person Re-identificationKnowledge Distillation	CodeCode Available	0
Weakly-Supervised Domain Adaptation of Deep Regression Trackers via Reinforced Knowledge Distillation	Mar 26, 2021	Domain AdaptationKnowledge Distillation	—Unverified	0
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation	Mar 25, 2021	Domain AdaptationKnowledge Distillation	CodeCode Available	1
Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data	Mar 25, 2021	Autonomous DrivingFew-Shot Learning	—Unverified	0
The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures	Mar 23, 2021	Information RetrievalKnowledge Distillation	—Unverified	0
Student Network Learning via Evolutionary Knowledge Distillation	Mar 23, 2021	Knowledge DistillationTransfer Learning	—Unverified	0
Balanced softmax cross-entropy for incremental learning with and without memory	Mar 23, 2021	class-incremental learningClass Incremental Learning	—Unverified	0
ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques	Mar 21, 2021	Knowledge Distillation	CodeCode Available	1
Compacting Deep Neural Networks for Internet of Things: Methods and Applications	Mar 20, 2021	DiversityKnowledge Distillation	—Unverified	0
Variational Knowledge Distillation for Disease Classification in Chest X-Rays	Mar 19, 2021	ClassificationGeneral Classification	—Unverified	0
Online Lifelong Generalized Zero-Shot Learning	Mar 19, 2021	Continual LearningGeneralized Zero-Shot Learning	CodeCode Available	0
Cost-effective Deployment of BERT Models in Serverless Environment	Mar 19, 2021	Knowledge DistillationSemantic Textual Similarity	—Unverified	0
Self-Supervised Adaptation for Video Super-Resolution	Mar 18, 2021	Image Super-ResolutionKnowledge Distillation	CodeCode Available	1
Human-Inspired Multi-Agent Navigation using Knowledge Distillation	Mar 18, 2021	Collision AvoidanceKnowledge Distillation	CodeCode Available	1
Similarity Transfer for Knowledge Distillation	Mar 18, 2021	Knowledge Distillation	—Unverified	0
Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation	Mar 17, 2021	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition	Mar 16, 2021	Deep LearningEmotion Recognition	—Unverified	0
Robustly Optimized and Distilled Training for Natural Language Understanding	Mar 16, 2021	Knowledge DistillationMachine Reading Comprehension	—Unverified	0
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation	Mar 15, 2021	Data AugmentationKnowledge Distillation	CodeCode Available	1
Robust Model Compression Using Deep Hypotheses	Mar 13, 2021	Binary ClassificationKnowledge Distillation	CodeCode Available	0
A New Training Framework for Deep Neural Network	Mar 12, 2021	Knowledge Distillation	—Unverified	0
Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones	Mar 10, 2021	Knowledge Distillationobject-detection	CodeCode Available	1

Show:10 25 50

← PrevPage 140 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified