Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3501–3525 of 4240 papers

Title	Date	Tasks	Status	Hype
Parser-Free Virtual Try-on via Distilling Appearance Flows	Mar 8, 2021	Human ParsingKnowledge Distillation	CodeCode Available	1
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning	Mar 6, 2021	class-incremental learningClass Incremental Learning	—Unverified	0
Adaptive Multi-Teacher Multi-level Knowledge Distillation	Mar 6, 2021	Knowledge Distillation	CodeCode Available	1
Teachers Do More Than Teach: Compressing Image-to-Image Models	Mar 5, 2021	Knowledge Distillation	CodeCode Available	1
Distributed Dynamic Map Fusion via Federated Learning for Intelligent Networked Vehicles	Mar 5, 2021	Federated LearningKnowledge Distillation	CodeCode Available	1
Deep Neural Network Models Compression	Mar 4, 2021	Knowledge DistillationQuantization	—Unverified	0
Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework	Mar 4, 2021	Knowledge DistillationNode Classification	CodeCode Available	1
General Instance Distillation for Object Detection	Mar 3, 2021	Knowledge DistillationModel Compression	CodeCode Available	1
Feature-Align Network with Knowledge Distillation for Efficient Denoising	Mar 2, 2021	DecoderDenoising	—Unverified	0
Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning	Mar 1, 2021	Few-Shot Image ClassificationFew-Shot Learning	CodeCode Available	1
Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network	Mar 1, 2021	Dynamic neural networksKnowledge Distillation	—Unverified	0
Training Generative Adversarial Networks in One Stage	Feb 28, 2021	Data-free Knowledge DistillationImage Generation	CodeCode Available	1
Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition	Feb 28, 2021	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Distilling Knowledge via Intermediate Classifiers	Feb 28, 2021	Knowledge DistillationTransfer Learning	CodeCode Available	1
PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation	Feb 26, 2021	ClusteringKnowledge Distillation	—Unverified	0
Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks	Feb 26, 2021	Computational EfficiencyKnowledge Distillation	—Unverified	0
Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation	Feb 25, 2021	Knowledge DistillationSelf-Knowledge Distillation	CodeCode Available	1
Localization Distillation for Dense Object Detection	Feb 24, 2021	Dense Object DetectionKnowledge Distillation	CodeCode Available	1
Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation	Feb 23, 2021	Knowledge Distillation	—Unverified	0
Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation	Feb 22, 2021	Dialogue GenerationGeneral Knowledge	—Unverified	0
CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray Segmentation	Feb 21, 2021	Image SegmentationKnowledge Distillation	CodeCode Available	1
Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification	Feb 20, 2021	Knowledge DistillationTransfer Learning	—Unverified	0
End-to-End Automatic Speech Recognition with Deep Mutual Learning	Feb 16, 2021	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation	Feb 16, 2021	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation	Feb 15, 2021	ClassificationGeneral Classification	—Unverified	0

Show:10 25 50

← PrevPage 141 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified