Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2326–2350 of 4240 papers

Title	Date	Tasks	Status	Hype
Curriculum Temperature for Knowledge Distillation	Nov 29, 2022	Image ClassificationKnowledge Distillation	CodeCode Available	1
SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification	Nov 28, 2022	Few-Shot Image ClassificationFew-Shot Learning	CodeCode Available	0
BJTU-WeChat's Systems for the WMT22 Chat Translation Task	Nov 28, 2022	DenoisingKnowledge Distillation	—Unverified	0
Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition	Nov 28, 2022	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation	Nov 28, 2022	Anomaly DetectionKnowledge Distillation	CodeCode Available	0
Dense Interspecies Face Embedding	Nov 28, 2022	Image ManipulationInterspecies Facial Keypoint Transfer	CodeCode Available	1
Class-aware Information for Logit-based Knowledge Distillation	Nov 27, 2022	Knowledge Distillation	—Unverified	0
Unbiased Knowledge Distillation for Recommendation	Nov 27, 2022	Knowledge DistillationModel Compression	CodeCode Available	1
EPIK: Eliminating multi-model Pipelines with Knowledge-distillation	Nov 27, 2022	Knowledge DistillationTransliteration	—Unverified	0
SKDBERT: Compressing BERT via Stochastic Knowledge Distillation	Nov 26, 2022	Knowledge DistillationLanguage Modeling	—Unverified	0
XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning	Nov 25, 2022	Action ClassificationClassification	CodeCode Available	1
Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding	Nov 25, 2022	3D visual groundingKnowledge Distillation	CodeCode Available	1
MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention	Nov 25, 2022	Knowledge DistillationNeural Architecture Search	CodeCode Available	1
Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment	Nov 23, 2022	Knowledge DistillationRepresentation Learning	CodeCode Available	1
Structural Knowledge Distillation for Object Detection	Nov 23, 2022	Feature ImportanceKnowledge Distillation	—Unverified	0
Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket	Nov 23, 2022	Data AugmentationKnowledge Distillation	CodeCode Available	1
DGEKT: A Dual Graph Ensemble Learning Method for Knowledge Tracing	Nov 23, 2022	Ensemble LearningKnowledge Distillation	CodeCode Available	1
Backdoor Cleansing with Unlabeled Data	Nov 22, 2022	Knowledge Distillation	CodeCode Available	1
On the Transferability of Visual Features in Generalized Zero-Shot Learning	Nov 22, 2022	Generalized Zero-Shot LearningKnowledge Distillation	CodeCode Available	0
Blind Knowledge Distillation for Robust Image Classification	Nov 21, 2022	Classificationimage-classification	CodeCode Available	0
Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version)	Nov 21, 2022	Knowledge DistillationMembership Inference Attack	CodeCode Available	0
Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation	Nov 21, 2022	Click-Through Rate PredictionKnowledge Distillation	CodeCode Available	1
Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text	Nov 21, 2022	Knowledge DistillationLanguage Modeling	CodeCode Available	1
AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation	Nov 20, 2022	Knowledge DistillationSelf-Knowledge Distillation	—Unverified	0
Scalable Collaborative Learning via Representation Sharing	Nov 20, 2022	Federated LearningKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 94 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified