Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 4176–4200 of 4240 papers

Title	Date	Tasks	Status	Hype
MotherNets: Rapid Deep Ensemble Learning	Sep 12, 2018	ClusteringClustering Ensemble	—Unverified	0
Label Denoising with Large Ensembles of Heterogeneous Neural Networks	Sep 12, 2018	Data AugmentationDenoising	—Unverified	0
RDPD: Rich Data Helps Poor Data via Imitation	Sep 6, 2018	Knowledge Distillation	CodeCode Available	0
Lifelong Learning via Progressive Distillation and Retrospection	Sep 1, 2018	Knowledge DistillationLifelong learning	—Unverified	0
Attention-Guided Answer Distillation for Machine Reading Comprehension	Aug 23, 2018	Knowledge DistillationMachine Reading Comprehension	—Unverified	0
Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks	Aug 17, 2018	Data AugmentationKnowledge Distillation	—Unverified	0
Cooperative Denoising for Distantly Supervised Relation Extraction	Aug 1, 2018	DenoisingInformation Retrieval	—Unverified	0
SlimNets: An Exploration of Deep Model Compression and Acceleration	Aug 1, 2018	Knowledge DistillationModel Compression	CodeCode Available	0
Self-supervised Knowledge Distillation Using Singular Value Decomposition	Jul 18, 2018	Knowledge DistillationTransfer Learning	CodeCode Available	0
Revisiting Distillation and Incremental Classifier Learning	Jul 8, 2018	Incremental LearningKnowledge Distillation	CodeCode Available	0
Distillation Techniques for Pseudo-rehearsal Based Incremental Learning	Jul 8, 2018	Incremental LearningKnowledge Distillation	CodeCode Available	0
Gradient Adversarial Training of Neural Networks	Jun 21, 2018	BIG-bench Machine LearningBinary Classification	—Unverified	0
Knowledge Distillation by On-the-Fly Native Ensemble	Jun 12, 2018	Computational Efficiencyimage-classification	CodeCode Available	0
Coupled End-to-End Transfer Learning With Generalized Fisher Information	Jun 1, 2018	DecoderDomain Adaptation	—Unverified	0
Collaborative Learning for Deep Neural Networks	May 30, 2018	Knowledge DistillationMulti-Task Learning	—Unverified	0
Channel Gating Neural Networks	May 29, 2018	Knowledge DistillationNetwork Pruning	CodeCode Available	1
A novel channel pruning method for deep neural network compression	May 29, 2018	channel selectionCombinatorial Optimization	—Unverified	0
Theory and Experiments on Vector Quantized Autoencoders	May 28, 2018	Image GenerationKnowledge Distillation	CodeCode Available	0
Visual Relationship Detection Based on Guided Proposals and Semantic Knowledge Distillation	May 28, 2018	Common Sense ReasoningKnowledge Distillation	—Unverified	0
Recurrent knowledge distillation	May 18, 2018	Knowledge Distillation	—Unverified	0
Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students	May 15, 2018	General Classificationimage-classification	—Unverified	0
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary	May 15, 2018	Adversarial AttackKnowledge Distillation	CodeCode Available	0
Born Again Neural Networks	May 12, 2018	Image ClassificationKnowledge Distillation	CodeCode Available	0
Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems	May 1, 2018	Knowledge DistillationRetrieval	CodeCode Available	0
Neural Compatibility Modeling with Attentive Knowledge Distillation	Apr 17, 2018	image-classificationImage Classification	—Unverified	0

Show:10 25 50

← PrevPage 168 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified