Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 901–925 of 4240 papers

Title	Date	Tasks	Status	Hype
Language Model Prior for Low-Resource Neural Machine Translation	Apr 30, 2020	Knowledge DistillationLanguage Modeling	CodeCode Available	1
Distilling Knowledge from Refinement in Multiple Instance Detection Networks	Apr 23, 2020	Knowledge DistillationMultiple Instance Learning	CodeCode Available	1
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation	Apr 21, 2020	Knowledge DistillationSentence	CodeCode Available	1
Role-Wise Data Augmentation for Knowledge Distillation	Apr 19, 2020	Data AugmentationKnowledge Distillation	CodeCode Available	1
Triplet Loss for Knowledge Distillation	Apr 17, 2020	Knowledge DistillationMetric Learning	CodeCode Available	1
Multimodal and multiview distillation for real-time player detection on a football field	Apr 16, 2020	Data AugmentationKnowledge Distillation	CodeCode Available	1
Dark Experience for General Continual Learning: a Strong, Simple Baseline	Apr 15, 2020	class-incremental learningClass Incremental Learning	CodeCode Available	1
Inter-Region Affinity Distillation for Road Marking Segmentation	Apr 11, 2020	Knowledge DistillationLane Detection	CodeCode Available	1
KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflow	Apr 11, 2020	CPUGPU	CodeCode Available	1
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling	Apr 8, 2020	Aspect ExtractionKnowledge Distillation	CodeCode Available	1
On the Effect of Dropping Layers of Pre-trained Transformer Models	Apr 8, 2020	Knowledge DistillationSentence	CodeCode Available	1
Towards Efficient Unconstrained Palmprint Recognition via Deep Distillation Hashing	Apr 7, 2020	Knowledge Distillation	CodeCode Available	1
Temporally Distributed Networks for Fast Video Semantic Segmentation	Apr 3, 2020	Knowledge DistillationReal-Time Semantic Segmentation	CodeCode Available	1
More Grounded Image Captioning by Distilling Image-Text Matching Model	Apr 1, 2020	Image CaptioningImage-text matching	CodeCode Available	1
Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing	Apr 1, 2020	Knowledge DistillationRetrieval	CodeCode Available	1
Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model	Mar 31, 2020	Active LearningKnowledge Distillation	CodeCode Available	1
Distilled Semantics for Comprehensive Scene Understanding from Videos	Mar 31, 2020	Depth EstimationKnowledge Distillation	CodeCode Available	1
Regularizing Class-wise Predictions via Self-knowledge Distillation	Mar 31, 2020	image-classificationImage Classification	CodeCode Available	1
Circumventing Outliers of AutoAugment with Knowledge Distillation	Mar 25, 2020	Data AugmentationGeneral Classification	CodeCode Available	1
Distilling Knowledge from Graph Convolutional Networks	Mar 23, 2020	Knowledge DistillationTransfer Learning	CodeCode Available	1
Collaborative Distillation for Ultra-Resolution Universal Style Transfer	Mar 18, 2020	DecoderGPU	CodeCode Available	1
Incremental Object Detection via Meta-Learning	Mar 17, 2020	Incremental LearningKnowledge Distillation	CodeCode Available	1
Deformation Flow Based Two-Stream Network for Lip Reading	Mar 12, 2020	Knowledge DistillationLipreading	CodeCode Available	1
SuperMix: Supervising the Mixing Data Augmentation	Mar 10, 2020	Data AugmentationGeneral Classification	CodeCode Available	1
Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN	Mar 9, 2020	Incremental LearningKnowledge Distillation	CodeCode Available	1

Show:10 25 50

← PrevPage 37 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified