Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 676–700 of 4240 papers

Title	Date	Tasks	Status	Hype
Up to 100 Faster Data-free Knowledge Distillation	Dec 12, 2021	Data-free Knowledge DistillationKnowledge Distillation	CodeCode Available	1
DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings	Dec 10, 2021	Contrastive LearningKnowledge Distillation	CodeCode Available	1
Mask-invariant Face Recognition through Template-level Knowledge Distillation	Dec 10, 2021	Face RecognitionKnowledge Distillation	CodeCode Available	1
Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation	Dec 7, 2021	Knowledge DistillationMulti-Task Learning	CodeCode Available	1
A Contrastive Distillation Approach for Incremental Semantic Segmentation in Aerial Images	Dec 7, 2021	image-classificationImage Classification	CodeCode Available	1
Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation	Dec 2, 2021	Knowledge DistillationNatural Language Understanding	CodeCode Available	1
A Fast Knowledge Distillation Framework for Visual Recognition	Dec 2, 2021	image-classificationImage Classification	CodeCode Available	1
Information Theoretic Representation Distillation	Dec 1, 2021	Classification with Binary Weight NetworkKnowledge Distillation	CodeCode Available	1
Aligned Structured Sparsity Learning for Efficient Image Super-Resolution	Dec 1, 2021	Image Super-ResolutionKnowledge Distillation	CodeCode Available	1
Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation	Dec 1, 2021	Anomaly DetectionGraph Neural Network	CodeCode Available	1
Distilling Meta Knowledge on Heterogeneous Graph for Illicit Drug Trafficker Detection on Social Media	Dec 1, 2021	Knowledge DistillationMarketing	CodeCode Available	1
Comprehensive Knowledge Distillation with Causal Intervention	Dec 1, 2021	Causal InferenceKnowledge Distillation	CodeCode Available	1
The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image	Dec 1, 2021	Knowledge Distillation	CodeCode Available	1
WiFi-based Multi-task Sensing	Nov 26, 2021	Gesture RecognitionIndoor Localization	CodeCode Available	1
Self-slimmed Vision Transformer	Nov 24, 2021	Knowledge Distillation	CodeCode Available	1
EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation	Nov 24, 2021	Event-based Object SegmentationKnowledge Distillation	CodeCode Available	1
Focal and Global Knowledge Distillation for Detectors	Nov 23, 2021	image-classificationImage Classification	CodeCode Available	1
On Representation Knowledge Distillation for Graph Neural Networks	Nov 9, 2021	Contrastive LearningKnowledge Distillation	CodeCode Available	1
Distilling Object Detectors with Feature Richness	Nov 1, 2021	Knowledge DistillationModel Compression	CodeCode Available	1
Learning Distilled Collaboration Graph for Multi-Agent Perception	Nov 1, 2021	3D Object DetectionKnowledge Distillation	CodeCode Available	1
Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data	Oct 27, 2021	Knowledge DistillationSemantic Segmentation	CodeCode Available	1
Instance-Conditional Knowledge Distillation for Object Detection	Oct 25, 2021	Image ClassificationKnowledge Distillation	CodeCode Available	1
Anti-Distillation Backdoor Attacks: Backdoors Can Really Survive in Knowledge Distillation	Oct 24, 2021	Backdoor AttackKnowledge Distillation	CodeCode Available	1
Pixel-by-Pixel Cross-Domain Alignment for Few-Shot Semantic Segmentation	Oct 22, 2021	Autonomous DrivingCross-Domain Few-Shot	CodeCode Available	1
Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation	Oct 17, 2021	Knowledge DistillationNode Classification	CodeCode Available	1

Show:10 25 50

← PrevPage 28 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified