Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3401–3425 of 4240 papers

Title	Date	Tasks	Status	Hype
Contrastive Model Inversion for Data-Free Knowledge Distillation	May 18, 2021	Contrastive LearningData-free Knowledge Distillation	CodeCode Available	1
Class-Incremental Few-Shot Object Detection	May 17, 2021	ClusteringFew-Shot Object Detection	—Unverified	0
Undistillable: Making A Nasty Teacher That CANNOT teach students	May 16, 2021	Knowledge Distillation	CodeCode Available	1
Graph-Free Knowledge Distillation for Graph Neural Networks	May 16, 2021	Knowledge DistillationTransfer Learning	CodeCode Available	1
AgeFlow: Conditional Age Progression and Regression with Normalizing Flows	May 15, 2021	AttributeKnowledge Distillation	CodeCode Available	1
Boosting Light-Weight Depth Estimation Via Knowledge Distillation	May 13, 2021	Computational EfficiencyDepth Estimation	CodeCode Available	1
When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks	May 13, 2021	Knowledge DistillationPose Estimation	CodeCode Available	1
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation	May 12, 2021	Adversarial TextData Augmentation	CodeCode Available	1
Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders	May 12, 2021	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation	May 10, 2021	Knowledge DistillationMixture-of-Experts	—Unverified	0
Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation	May 8, 2021	DenoisingKnowledge Distillation	—Unverified	0
Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates	May 7, 2021	Knowledge Distillationmodel	—Unverified	0
Initialization and Regularization of Factorized Neural Layers	May 3, 2021	Knowledge DistillationModel Compression	CodeCode Available	1
Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack	May 3, 2021	Knowledge DistillationSelf-Knowledge Distillation	—Unverified	0
Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias	May 1, 2021	Knowledge DistillationMachine Translation	CodeCode Available	0
Knowledge Distillation for Swedish NER models: A Search for Performance and Efficiency	May 1, 2021	Knowledge DistillationModel Compression	—Unverified	0
A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts	May 1, 2021	Explainable artificial intelligenceKnowledge Distillation	—Unverified	0
Distilling EEG Representations via Capsules for Affective Computing	Apr 30, 2021	EEGElectroencephalogram (EEG)	—Unverified	0
Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation	Apr 30, 2021	Image-to-Image TranslationKnowledge Distillation	—Unverified	0
Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer	Apr 29, 2021	General KnowledgeKnowledge Distillation	—Unverified	0
LIDAR and Position-Aided mmWave Beam Selection with Non-local CNNs and Curriculum Training	Apr 29, 2021	Knowledge DistillationPosition	CodeCode Available	0
Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network	Apr 28, 2021	Graph Neural NetworkKnowledge Distillation	CodeCode Available	0
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation	Apr 28, 2021	image-classificationImage Classification	CodeCode Available	1
Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification	Apr 27, 2021	ClassificationGeneral Classification	—Unverified	0
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation	Apr 24, 2021	Knowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 137 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified