Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2626–2650 of 4240 papers

Title	Date	Tasks	Status	Hype
Knowledge Condensation Distillation	Jul 12, 2022	Knowledge Distillation	CodeCode Available	1
HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors	Jul 12, 2022	Knowledge DistillationObject	CodeCode Available	1
Cross-Architecture Knowledge Distillation	Jul 12, 2022	Knowledge Distillation	—Unverified	0
Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis	Jul 11, 2022	GPUKnowledge Distillation	CodeCode Available	1
2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds	Jul 10, 2022	3D Semantic SegmentationAutonomous Driving	CodeCode Available	2
1st Place Solution to the EPIC-Kitchens Action Anticipation Challenge 2022	Jul 10, 2022	Action AnticipationKnowledge Distillation	—Unverified	0
FairDistillation: Mitigating Stereotyping in Language Models	Jul 10, 2022	Knowledge Distillation	CodeCode Available	1
Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies	Jul 6, 2022	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Low-resource Low-footprint Wake-word Detection using Knowledge Distillation	Jul 6, 2022	Knowledge Distillationspeech-recognition	—Unverified	0
PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient	Jul 5, 2022	Knowledge Distillationobject-detection	—Unverified	0
GLANCE: Global to Local Architecture-Neutral Concept-based Explanations	Jul 5, 2022	DisentanglementFeature Importance	CodeCode Available	0
Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer	Jul 5, 2022	Image-text matchingKnowledge Distillation	CodeCode Available	1
ACT-Net: Asymmetric Co-Teacher Network for Semi-supervised Memory-efficient Medical Image Segmentation	Jul 5, 2022	Image SegmentationKnowledge Distillation	CodeCode Available	0
A Generative Framework for Personalized Learning and Estimation: Theory, Algorithms, and Privacy	Jul 5, 2022	Federated LearningKnowledge Distillation	—Unverified	0
VEM^2L: A Plug-and-play Framework for Fusing Text and Structure Knowledge on Sparse Knowledge Graph Completion	Jul 4, 2022	Knowledge DistillationKnowledge Graph Completion	—Unverified	0
FasterAI: A Lightweight Library for Creating Sparse Neural Networks	Jul 3, 2022	Knowledge Distillation	—Unverified	0
PrUE: Distilling Knowledge from Sparse Teacher Networks	Jul 3, 2022	Knowledge Distillation	CodeCode Available	0
Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation	Jul 2, 2022	Knowledge DistillationMulti-Task Learning	—Unverified	0
Lost in Distillation: A Case Study in Toxicity Modeling	Jul 1, 2022	Knowledge Distillation	—Unverified	0
Asynchronous Convergence in Multi-Task Learning via Knowledge Distillation from Converged Tasks	Jul 1, 2022	Knowledge DistillationMulti-Task Learning	—Unverified	0
KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation	Jul 1, 2022	Knowledge DistillationLanguage Modeling	—Unverified	0
Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT	Jul 1, 2022	Knowledge Distillation	—Unverified	0
End-to-End Simultaneous Speech Translation with Pretraining and Distillation: Huawei Noah’s System for AutoSimTranS 2022	Jul 1, 2022	DecoderKnowledge Distillation	—Unverified	0
FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning	Jul 1, 2022	Knowledge DistillationPhoneme Recognition	CodeCode Available	1
ListBERT: Learning to Rank E-commerce products with Listwise BERT	Jun 30, 2022	Knowledge DistillationLearning-To-Rank	—Unverified	0

Show:10 25 50

← PrevPage 106 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified