Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 851–875 of 4240 papers

Title	Date	Tasks	Status	Hype
Simplified TinyBERT: Knowledge Distillation for Document Retrieval	Sep 16, 2020	Document RankingKnowledge Distillation	CodeCode Available	1
Noisy Self-Knowledge Distillation for Text Summarization	Sep 15, 2020	Knowledge DistillationSelf-Knowledge Distillation	CodeCode Available	1
Simulating Unknown Target Models for Query-Efficient Black-box Attacks	Sep 2, 2020	Knowledge DistillationMeta-Learning	CodeCode Available	1
Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition	Sep 1, 2020	Action RecognitionImage Generation	CodeCode Available	1
Unpaired Learning of Deep Image Denoising	Aug 31, 2020	DenoisingImage Denoising	CodeCode Available	1
Performance Optimization for Federated Person Re-identification via Benchmark Analysis	Aug 26, 2020	Federated LearningKnowledge Distillation	CodeCode Available	1
PARADE: Passage Representation Aggregation for Document Reranking	Aug 20, 2020	Ad-Hoc Information RetrievalDocument Ranking	CodeCode Available	1
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation	Aug 18, 2020	Knowledge DistillationRepresentation Learning	CodeCode Available	1
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR	Aug 9, 2020	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	CodeCode Available	1
Improving Knowledge Distillation via Category Structure	Aug 1, 2020	Knowledge Distillation	CodeCode Available	1
Intra-class Feature Variation Distillation for Semantic Segmentation	Aug 1, 2020	Knowledge DistillationSegmentation	CodeCode Available	1
Distilling Visual Priors from Self-Supervised Learning	Aug 1, 2020	ClassificationContrastive Learning	CodeCode Available	1
Weakly Supervised 3D Object Detection from Point Clouds	Jul 28, 2020	3D Object DetectionKnowledge Distillation	CodeCode Available	1
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge	Jul 28, 2020	Federated LearningKnowledge Distillation	CodeCode Available	1
Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation	Jul 21, 2020	Instance SegmentationKnowledge Distillation	CodeCode Available	1
Resolution Switchable Networks for Runtime Efficient Image Recognition	Jul 19, 2020	Knowledge DistillationQuantization	CodeCode Available	1
Self-supervision on Unlabelled OR Data for Multi-person 2D/3D Human Pose Estimation	Jul 16, 2020	3D Human Pose Estimation3D Pose Estimation	CodeCode Available	1
Defocus Blur Detection via Depth Distillation	Jul 16, 2020	DecoderDefocus Blur Detection	CodeCode Available	1
Knowledge Distillation for Multi-task Learning	Jul 14, 2020	Knowledge DistillationMulti-Task Learning	CodeCode Available	1
Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation	Jul 14, 2020	Domain AdaptationKnowledge Distillation	CodeCode Available	1
Learning to Learn Parameterized Classification Networks for Scalable Input Images	Jul 13, 2020	ClassificationGeneral Classification	CodeCode Available	1
Towards Practical Lipreading with Distilled and Efficient Models	Jul 13, 2020	Knowledge DistillationLipreading	CodeCode Available	1
Temporal Self-Ensembling Teacher for Semi-Supervised Object Detection	Jul 13, 2020	image-classificationImage Classification	CodeCode Available	1
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning	Jul 13, 2020	Continual LearningImage Captioning	CodeCode Available	1
Robust Re-Identification by Multiple Views Knowledge Distillation	Jul 8, 2020	Knowledge DistillationPerson Re-Identification	CodeCode Available	1

Show:10 25 50

← PrevPage 35 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified