Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2576–2600 of 4240 papers

Title	Date	Tasks	Status	Hype
PGX: A Multi-level GNN Explanation Framework Based on Separate Knowledge Distillation Processes	Aug 5, 2022	Knowledge DistillationRepresentation Learning	—Unverified	0
Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Prediction	Aug 4, 2022	Knowledge DistillationPrediction	CodeCode Available	1
Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection	Aug 4, 2022	Diabetic Retinopathy DetectionKnowledge Distillation	—Unverified	0
KD-SCFNet: Towards More Accurate and Efficient Salient Object Detection via Knowledge Distillation	Aug 3, 2022	Knowledge Distillationobject-detection	CodeCode Available	1
Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer	Aug 1, 2022	Activity RecognitionContrastive Learning	—Unverified	0
Generative Bias for Robust Visual Question Answering	Aug 1, 2022	Knowledge DistillationQuestion Answering	CodeCode Available	1
Chinese grammatical error correction based on knowledge distillation	Jul 31, 2022	Grammatical Error CorrectionKnowledge Distillation	CodeCode Available	1
Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval	Jul 31, 2022	Knowledge DistillationLanguage Modeling	CodeCode Available	1
SDBERT: SparseDistilBERT, a faster and smaller BERT model	Jul 28, 2022	Knowledge Distillation	—Unverified	0
Meta-Learning based Degradation Representation for Blind Super-Resolution	Jul 28, 2022	Blind Super-ResolutionKnowledge Distillation	CodeCode Available	1
NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation	Jul 27, 2022	Graph GenerationKnowledge Distillation	—Unverified	0
Exploring Generalizable Distillation for Efficient Medical Image Segmentation	Jul 26, 2022	DecoderImage Segmentation	CodeCode Available	0
HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks	Jul 25, 2022	Knowledge DistillationVocal Bursts Intensity Prediction	—Unverified	0
Domain-invariant Feature Exploration for Domain Generalization	Jul 25, 2022	DiversityDomain Generalization	—Unverified	0
Black-box Few-shot Knowledge Distillation	Jul 25, 2022	image-classificationImage Classification	CodeCode Available	1
Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations	Jul 25, 2022	Few-Shot Object DetectionKnowledge Distillation	—Unverified	0
Spatial-Channel Token Distillation for Vision MLPs	Jul 23, 2022	Image ClassificationKnowledge Distillation	CodeCode Available	0
Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion	Jul 23, 2022	Data-free Knowledge DistillationFairness	CodeCode Available	0
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition	Jul 23, 2022	Contrastive Learningimage-classification	CodeCode Available	1
Hyper-Representations for Pre-Training and Transfer Learning	Jul 22, 2022	Knowledge DistillationNeural Architecture Search	CodeCode Available	1
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay	Jul 22, 2022	class-incremental learningClass Incremental Learning	CodeCode Available	0
Federated Semi-Supervised Domain Adaptation via Knowledge Transfer	Jul 21, 2022	Domain AdaptationFederated Learning	—Unverified	0
TinyViT: Fast Pretraining Distillation for Small Vision Transformers	Jul 21, 2022	Image ClassificationKnowledge Distillation	—Unverified	0
KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo	Jul 21, 2022	Knowledge DistillationSelf-Supervised Learning	CodeCode Available	1
Many-to-One Knowledge Distillation of Real-Time Epileptic Seizure Detection for Low-Power Wearable Internet of Things Systems	Jul 20, 2022	Edge-computingKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 104 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified