Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1976–2000 of 4240 papers

Title	Date	Tasks	Status	Hype
Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty	May 4, 2023	Knowledge Distillationobject-detection	CodeCode Available	1
SCOTT: Self-Consistent Chain-of-Thought Distillation	May 3, 2023	counterfactualCounterfactual Reasoning	CodeCode Available	1
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training	May 3, 2023	Knowledge DistillationText Generation	CodeCode Available	0
DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge Distillation	May 2, 2023	Knowledge DistillationSemantic Segmentation	CodeCode Available	1
Structure Aware Incremental Learning with Personalized Imitation Weights for Recommender Systems	May 2, 2023	Incremental LearningKnowledge Distillation	—Unverified	0
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models	May 2, 2023	Knowledge Distillation	—Unverified	0
Detect, Distill and Update: Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data	May 1, 2023	Knowledge DistillationSynthetic Data Generation	CodeCode Available	0
Refined Response Distillation for Class-Incremental Player Detection	May 1, 2023	Knowledge Distillationobject-detection	CodeCode Available	0
Scaffolding a Student to Instill Knowledge	May 1, 2023	Knowledge Distillation	CodeCode Available	0
Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation	Apr 28, 2023	Knowledge DistillationSemantic Segmentation	CodeCode Available	0
CORSD: Class-Oriented Relational Self Distillation	Apr 28, 2023	Knowledge DistillationModel Compression	—Unverified	0
Ensemble Modeling with Contrastive Knowledge Distillation for Sequential Recommendation	Apr 28, 2023	AttributeContrastive Learning	CodeCode Available	0
Learning Human-Human Interactions in Images from Weak Textual Supervision	Apr 27, 2023	Human-Human Interaction RecognitionImage Captioning	—Unverified	0
A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering	Apr 26, 2023	DecoderKnowledge Distillation	CodeCode Available	1
Shape-Net: Room Layout Estimation from Panoramic Images Robust to Occlusion using Knowledge Distillation with 3D Shapes as Additional Inputs	Apr 25, 2023	3D geometry3D Reconstruction	—Unverified	0
Class Attention Transfer Based Knowledge Distillation	Apr 25, 2023	Knowledge DistillationModel Compression	CodeCode Available	1
Improving Knowledge Distillation via Transferring Learning Ability	Apr 24, 2023	Knowledge Distillation	CodeCode Available	0
A Forward and Backward Compatible Framework for Few-shot Class-incremental Pill Recognition	Apr 24, 2023	class-incremental learningClass Incremental Learning	CodeCode Available	0
Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving	Apr 24, 2023	Autonomous DrivingAutonomous Vehicles	—Unverified	0
Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic Segmentation	Apr 22, 2023	Autonomous DrivingKnowledge Distillation	CodeCode Available	1
Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation	Apr 20, 2023	Knowledge DistillationMachine Translation	—Unverified	0
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs	Apr 20, 2023	Knowledge DistillationNode Classification	CodeCode Available	1
Word Sense Induction with Knowledge Distillation from BERT	Apr 20, 2023	Knowledge DistillationLanguage Modeling	—Unverified	0
Attention Weighted Local Descriptors	Apr 19, 2023	3D ReconstructionHomography Estimation	CodeCode Available	1
Knowledge Distillation Under Ideal Joint Classifier Assumption	Apr 19, 2023	Domain AdaptationKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 80 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified