Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2876–2900 of 4240 papers

Title	Date	Tasks	Status	Hype
Student Becomes Decathlon Master in Retinal Vessel Segmentation via Dual-teacher Multi-target Domain Adaptation	Mar 7, 2022	Domain AdaptationDomain Generalization	CodeCode Available	0
Enhance Language Identification using Dual-mode Model with Knowledge Distillation	Mar 7, 2022	Knowledge DistillationLanguage Identification	CodeCode Available	0
Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning	Mar 5, 2022	GPUKnowledge Distillation	CodeCode Available	1
Consistent Representation Learning for Continual Relation Extraction	Mar 5, 2022	Continual Relation ExtractionContrastive Learning	CodeCode Available	1
Better Supervisory Signals by Observing Learning Paths	Mar 4, 2022	Knowledge Distillation	CodeCode Available	0
MIAShield: Defending Membership Inference Attacks via Preemptive Exclusion of Members	Mar 2, 2022	image-classificationImage Classification	—Unverified	0
X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning	Mar 2, 2022	3D dense captioningDense Captioning	CodeCode Available	1
TRILLsson: Distilled Universal Paralinguistic Speech Representations	Mar 1, 2022	Emotion RecognitionKnowledge Distillation	—Unverified	0
Dual Embodied-Symbolic Concept Representations for Deep Learning	Mar 1, 2022	class-incremental learningClass Incremental Learning	—Unverified	0
Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology	Mar 1, 2022	DiversityKnowledge Distillation	CodeCode Available	1
Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation	Feb 28, 2022	DecoderKnowledge Distillation	—Unverified	0
TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation	Feb 27, 2022	Autonomous DrivingKnowledge Distillation	CodeCode Available	1
Content-Variant Reference Image Quality Assessment via Knowledge Distillation	Feb 26, 2022	Image Quality AssessmentKnowledge Distillation	CodeCode Available	1
Joint Answering and Explanation for Visual Commonsense Reasoning	Feb 25, 2022	Knowledge DistillationQuestion Answering	CodeCode Available	0
Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation	Feb 25, 2022	Knowledge DistillationPrediction	—Unverified	0
Learn From the Past: Experience Ensemble Knowledge Distillation	Feb 25, 2022	Knowledge DistillationTransfer Learning	—Unverified	0
Efficient Video Segmentation Models with Per-frame Inference	Feb 24, 2022	Image MattingInstance Segmentation	—Unverified	0
Are All Linear Regions Created Equal?	Feb 23, 2022	AllKnowledge Distillation	CodeCode Available	0
Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification	Feb 23, 2022	ClassificationIncremental Learning	—Unverified	0
Distilled Neural Networks for Efficient Learning to Rank	Feb 22, 2022	CPUInformation Retrieval	CodeCode Available	0
A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation	Feb 21, 2022	Knowledge DistillationModel Compression	—Unverified	0
Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning	Feb 21, 2022	Continual LearningKnowledge Distillation	—Unverified	0
CaMEL: Mean Teacher Learning for Image Captioning	Feb 21, 2022	Image CaptioningKnowledge Distillation	CodeCode Available	1
Cross-Task Knowledge Distillation in Multi-Task Recommendation	Feb 20, 2022	Knowledge DistillationMulti-Task Learning	—Unverified	0
General Cyclical Training of Neural Networks	Feb 17, 2022	Data AugmentationKnowledge Distillation	CodeCode Available	1

Show:10 25 50

← PrevPage 116 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified