Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3976–4000 of 4240 papers

Title	Date	Tasks	Status	Hype
Distributed Soft Actor-Critic with Multivariate Reward Representation and Knowledge Distillation	Nov 29, 2019	Knowledge Distillationreinforcement-learning	CodeCode Available	0
Towards Oracle Knowledge Distillation with Neural Architecture Search	Nov 29, 2019	image-classificationImage Classification	—Unverified	0
Blockwisely Supervised Neural Architecture Search with Knowledge Distillation	Nov 29, 2019	Knowledge DistillationNeural Architecture Search	CodeCode Available	1
QKD: Quantization-aware Knowledge Distillation	Nov 28, 2019	Knowledge DistillationQuantization	—Unverified	0
Data-Driven Compression of Convolutional Neural Networks	Nov 28, 2019	Knowledge DistillationModel Compression	—Unverified	0
Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers	Nov 26, 2019	Knowledge DistillationLipreading	—Unverified	0
Go From the General to the Particular: Multi-Domain Translation with Domain Transformation Networks	Nov 22, 2019	DecoderGeneral Knowledge	CodeCode Available	1
Few Shot Network Compression via Cross Distillation	Nov 21, 2019	Knowledge DistillationModel Compression	CodeCode Available	0
Search to Distill: Pearls are Everywhere but not the Eyes	Nov 20, 2019	Ensemble LearningFace Recognition	—Unverified	0
Neural Network Pruning with Residual-Connections and Limited-Data	Nov 19, 2019	Knowledge DistillationNetwork Pruning	CodeCode Available	0
Towards Making Deep Transfer Learning Never Hurt	Nov 18, 2019	AllKnowledge Distillation	—Unverified	0
Preparing Lessons: Improve Knowledge Distillation with Better Supervision	Nov 18, 2019	Knowledge Distillation	CodeCode Available	1
Maintaining Discrimination and Fairness in Class Incremental Learning	Nov 16, 2019	class-incremental learningClass Incremental Learning	CodeCode Available	1
Data Efficient Stagewise Knowledge Distillation	Nov 15, 2019	Knowledge DistillationModel Compression	CodeCode Available	0
Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation	Nov 13, 2019	Image ClassificationKnowledge Distillation	—Unverified	0
Learning from a Teacher using Unlabeled Data	Nov 13, 2019	Knowledge DistillationModel Compression	CodeCode Available	1
Collaborative Distillation for Top-N Recommendation	Nov 13, 2019	Collaborative FilteringKnowledge Distillation	—Unverified	0
Knowledge Distillation in Document Retrieval	Nov 11, 2019	Knowledge DistillationRetrieval	—Unverified	0
Graph Representation Learning via Multi-task Knowledge Distillation	Nov 11, 2019	Graph Representation LearningKnowledge Distillation	—Unverified	0
Scalable Zero-shot Entity Linking with Dense Entity Retrieval	Nov 10, 2019	Entity EmbeddingsEntity Linking	CodeCode Available	2
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models	Nov 9, 2019	Knowledge DistillationMulti-Task Learning	—Unverified	0
Knowledge Distillation for Incremental Learning in Semantic Segmentation	Nov 8, 2019	image-classificationImage Classification	—Unverified	0
Deep geometric knowledge distillation with graphs	Nov 8, 2019	Knowledge Distillation	CodeCode Available	0
Microsoft Research Asia's Systems for WMT19	Nov 7, 2019	Data AugmentationKnowledge Distillation	—Unverified	0
Teacher-Student Training for Robust Tacotron-based TTS	Nov 7, 2019	DecoderKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 160 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified