Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2226–2250 of 4240 papers

Title	Date	Tasks	Status	Hype
Designing an Improved Deep Learning-based Model for COVID-19 Recognition in Chest X-ray Images: A Knowledge Distillation Approach	Jan 6, 2023	Knowledge Distillation	—Unverified	0
Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation	Jan 3, 2023	BenchmarkingFew-shot Instance Segmentation	CodeCode Available	1
RELIANT: Fair Knowledge Distillation for Graph Neural Networks	Jan 3, 2023	FairnessGraph Learning	CodeCode Available	0
Knowledge-guided Causal Intervention for Weakly-supervised Object Localization	Jan 3, 2023	Knowledge DistillationObject	CodeCode Available	0
Label-Guided Knowledge Distillation for Continual Semantic Segmentation on 2D Images and 3D Point Clouds	Jan 1, 2023	Continual Semantic SegmentationKnowledge Distillation	CodeCode Available	1
Multi-Task Learning with Knowledge Distillation for Dense Prediction	Jan 1, 2023	Boundary DetectionDepth Estimation	—Unverified	0
Automated Knowledge Distillation via Monte Carlo Tree Search	Jan 1, 2023	image-classificationImage Classification	CodeCode Available	0
TripLe: Revisiting Pretrained Model Reuse and Progressive Learning for Efficient Vision Transformer Scaling and Searching	Jan 1, 2023	Knowledge DistillationNeural Architecture Search	—Unverified	0
Continual Segment: Towards a Single, Unified and Non-forgetting Continual Segmentation Model of 143 Whole-body Organs in CT Scans	Jan 1, 2023	Continual Semantic SegmentationDecoder	—Unverified	0
Knowledge-Spreader: Learning Semi-Supervised Facial Action Dynamics by Consistifying Knowledge Granularity	Jan 1, 2023	Knowledge Distillation	—Unverified	0
UniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors	Jan 1, 2023	Knowledge Distillation	—Unverified	0
Alleviating Catastrophic Forgetting of Incremental Object Detection via Within-Class and Between-Class Knowledge Distillation	Jan 1, 2023	Knowledge Distillationobject-detection	—Unverified	0
Remembering Normality: Memory-guided Knowledge Distillation for Unsupervised Anomaly Detection	Jan 1, 2023	Anomaly DetectionKnowledge Distillation	CodeCode Available	1
MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices	Jan 1, 2023	Efficient Neural NetworkImage Inpainting	CodeCode Available	2
Tiny Updater: Towards Efficient Neural Network-Driven Software Updating	Jan 1, 2023	Efficient Neural Networkimage-classification	CodeCode Available	0
Data-Free Class-Incremental Hand Gesture Recognition	Jan 1, 2023	class-incremental learningClass Incremental Learning	CodeCode Available	1
Distilling DETR with Visual-Linguistic Knowledge for Open-Vocabulary Object Detection	Jan 1, 2023	Knowledge DistillationLanguage Modeling	CodeCode Available	1
Masked Autoencoders Are Stronger Knowledge Distillers	Jan 1, 2023	DecoderKnowledge Distillation	—Unverified	0
Dual Learning with Dynamic Knowledge Distillation for Partially Relevant Video Retrieval	Jan 1, 2023	Knowledge DistillationLanguage Modelling	CodeCode Available	1
ICD-Face: Intra-class Compactness Distillation for Face Recognition	Jan 1, 2023	Face RecognitionKnowledge Distillation	—Unverified	0
Beyond the Limitation of Monocular 3D Detector via Knowledge Distillation	Jan 1, 2023	Knowledge Distillation	CodeCode Available	0
Data-Free Knowledge Distillation via Feature Exchange and Activation Region Constraint	Jan 1, 2023	Data AugmentationData-free Knowledge Distillation	CodeCode Available	1
ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector	Jan 1, 2023	Knowledge Distillationobject-detection	—Unverified	0
Probabilistic Knowledge Distillation of Face Ensembles	Jan 1, 2023	Face Image QualityFace Recognition	—Unverified	0
Multi-Level Logit Distillation	Jan 1, 2023	Knowledge DistillationPrediction	CodeCode Available	1

Show:10 25 50

← PrevPage 90 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified