Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3101–3125 of 4240 papers

Title	Date	Tasks	Status	Hype
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding	Nov 1, 2021	Adversarial RobustnessAll	—Unverified	0
Limitations of Knowledge Distillation for Zero-shot Transfer Learning	Nov 1, 2021	CPUCross-Lingual Transfer	—Unverified	0
Distilling Object Detectors with Feature Richness	Nov 1, 2021	Knowledge DistillationModel Compression	CodeCode Available	1
Learning Distilled Collaboration Graph for Multi-Agent Perception	Nov 1, 2021	3D Object DetectionKnowledge Distillation	CodeCode Available	1
PP-ShiTu: A Practical Lightweight Image Recognition System	Nov 1, 2021	Face RecognitionKnowledge Distillation	CodeCode Available	0
Rethinking the Knowledge Distillation From the Perspective of Model Calibration	Oct 31, 2021	Knowledge Distillation	—Unverified	0
Estimating and Maximizing Mutual Information for Knowledge Distillation	Oct 29, 2021	Knowledge Distillation	—Unverified	0
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks	Oct 29, 2021	Knowledge DistillationModel Compression	—Unverified	0
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM	Oct 28, 2021	Knowledge DistillationNatural Language Understanding	—Unverified	0
Towards Model Agnostic Federated Learning Using Knowledge Distillation	Oct 28, 2021	Federated LearningKnowledge Distillation	—Unverified	0
Temporal Knowledge Distillation for On-device Audio Classification	Oct 27, 2021	Audio ClassificationClassification	—Unverified	0
GenURL: A General Framework for Unsupervised Representation Learning	Oct 27, 2021	Contrastive LearningDimensionality Reduction	—Unverified	0
Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data	Oct 27, 2021	Knowledge DistillationSemantic Segmentation	CodeCode Available	1
Beyond Classification: Knowledge Distillation using Multi-Object Impressions	Oct 27, 2021	ClassificationKnowledge Distillation	—Unverified	0
Response-based Distillation for Incremental Object Detection	Oct 26, 2021	Incremental LearningKnowledge Distillation	—Unverified	0
Instance-Conditional Knowledge Distillation for Object Detection	Oct 25, 2021	Image ClassificationKnowledge Distillation	CodeCode Available	1
Reconstructing Pruned Filters using Cheap Spatial Transformations	Oct 25, 2021	Feature CompressionKnowledge Distillation	—Unverified	0
MUSE: Feature Self-Distillation with Mutual Information and Self-Information	Oct 25, 2021	image-classificationImage Classification	—Unverified	0
Anti-Distillation Backdoor Attacks: Backdoors Can Really Survive in Knowledge Distillation	Oct 24, 2021	Backdoor AttackKnowledge Distillation	CodeCode Available	1
X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation	Oct 24, 2021	Depth EstimationKnowledge Distillation	—Unverified	0
How and When Adversarial Robustness Transfers in Knowledge Distillation?	Oct 22, 2021	Adversarial RobustnessKnowledge Distillation	—Unverified	0
Pseudo Supervised Monocular Depth Estimation with Teacher-Student Network	Oct 22, 2021	Depth EstimationKnowledge Distillation	—Unverified	0
Pixel-by-Pixel Cross-Domain Alignment for Few-Shot Semantic Segmentation	Oct 22, 2021	Autonomous DrivingCross-Domain Few-Shot	CodeCode Available	1
Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression	Oct 21, 2021	Knowledge DistillationModel Compression	—Unverified	0
Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach	Oct 20, 2021	Knowledge DistillationLanguage Modeling	—Unverified	0

Show:10 25 50

← PrevPage 125 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified