Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3001–3025 of 4240 papers

Title	Date	Tasks	Status	Hype
Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation	Dec 9, 2021	image-classificationImage Classification	—Unverified	0
Boosting Contrastive Learning with Relation Knowledge Distillation	Dec 8, 2021	Contrastive LearningKnowledge Distillation	—Unverified	0
Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation	Dec 7, 2021	Auxiliary LearningKnowledge Distillation	CodeCode Available	0
A Contrastive Distillation Approach for Incremental Semantic Segmentation in Aerial Images	Dec 7, 2021	image-classificationImage Classification	CodeCode Available	1
Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation	Dec 7, 2021	Knowledge DistillationMulti-Task Learning	CodeCode Available	1
ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Images	Dec 7, 2021	DeepFake DetectionFace Swapping	CodeCode Available	0
Safe Distillation Box	Dec 5, 2021	Knowledge Distillation	—Unverified	0
CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks	Dec 5, 2021	ClassificationContinual Learning	—Unverified	0
Extracting knowledge from features with multilevel abstraction	Dec 4, 2021	Data AugmentationKnowledge Distillation	—Unverified	0
KDCTime: Knowledge Distillation with Calibration on InceptionTime for Time-series Classification	Dec 4, 2021	Knowledge DistillationTime Series	—Unverified	0
Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation	Dec 2, 2021	Knowledge DistillationNatural Language Understanding	CodeCode Available	1
FedRAD: Federated Robust Adaptive Distillation	Dec 2, 2021	Federated LearningKnowledge Distillation	—Unverified	0
A Fast Knowledge Distillation Framework for Visual Recognition	Dec 2, 2021	image-classificationImage Classification	CodeCode Available	1
Information Theoretic Representation Distillation	Dec 1, 2021	Classification with Binary Weight NetworkKnowledge Distillation	CodeCode Available	1
The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image	Dec 1, 2021	Knowledge Distillation	CodeCode Available	1
Distilling Meta Knowledge on Heterogeneous Graph for Illicit Drug Trafficker Detection on Social Media	Dec 1, 2021	Knowledge DistillationMarketing	CodeCode Available	1
Aligned Structured Sparsity Learning for Efficient Image Super-Resolution	Dec 1, 2021	Image Super-ResolutionKnowledge Distillation	CodeCode Available	1
Shapeshifter: a Parameter-efficient Transformer using Factorized Reshaped Matrices	Dec 1, 2021	Knowledge DistillationModel Compression	CodeCode Available	0
Handling Long-tailed Feature Distribution in AdderNets	Dec 1, 2021	Knowledge Distillation	—Unverified	0
Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation	Dec 1, 2021	Anomaly DetectionGraph Neural Network	CodeCode Available	1
Comprehensive Knowledge Distillation with Causal Intervention	Dec 1, 2021	Causal InferenceKnowledge Distillation	CodeCode Available	1
Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation	Dec 1, 2021	Knowledge Distillation	CodeCode Available	0
Adversarial Teacher-Student Representation Learning for Domain Generalization	Dec 1, 2021	Data AugmentationDomain Generalization	CodeCode Available	0
Unsupervised Representation Transfer for Small Networks: I Believe I Can Distill On-the-Fly	Dec 1, 2021	Knowledge DistillationLinear evaluation	—Unverified	0
Using a GAN to Generate Adversarial Examples to Facial Image Recognition	Nov 30, 2021	Face RecognitionGenerative Adversarial Network	—Unverified	0

Show:10 25 50

← PrevPage 121 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified