Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2851–2875 of 4240 papers

Title	Date	Tasks	Status	Hype
SC2 Benchmark: Supervised Compression for Split Computing	Mar 16, 2022	Data CompressionEdge-computing	—Unverified	0
Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation	Mar 16, 2022	Image SegmentationKnowledge Distillation	CodeCode Available	1
Unified Visual Transformer Compression	Mar 15, 2022	Knowledge Distillation	CodeCode Available	1
SATS: Self-Attention Transfer for Continual Semantic Segmentation	Mar 15, 2022	Continual Semantic SegmentationKnowledge Distillation	CodeCode Available	1
On the benefits of knowledge distillation for adversarial robustness	Mar 14, 2022	Adversarial RobustnessKnowledge Distillation	—Unverified	0
DS3-Net: Difficulty-perceived Common-to-T1ce Semi-Supervised Multimodal MRI Synthesis Network	Mar 14, 2022	Knowledge DistillationSSIM	—Unverified	0
CEKD:Cross Ensemble Knowledge Distillation for Augmented Fine-grained Data	Mar 13, 2022	Data AugmentationKnowledge Distillation	—Unverified	0
CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification	Mar 13, 2022	Audio ClassificationKnowledge Distillation	CodeCode Available	3
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation	Mar 12, 2022	Image CaptioningKnowledge Distillation	—Unverified	0
Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation	Mar 12, 2022	Image-to-Image TranslationKnowledge Distillation	—Unverified	0
Medical Image Segmentation on MRI Images with Missing Modalities: A Review	Mar 11, 2022	Image GenerationImage Segmentation	—Unverified	0
Deep Class Incremental Learning from Decentralized Data	Mar 11, 2022	class-incremental learningClass Incremental Learning	CodeCode Available	0
Improving Neural ODEs via Knowledge Distillation	Mar 10, 2022	Knowledge Distillation	—Unverified	0
Look Backward and Forward: Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translation	Mar 10, 2022	DecoderKnowledge Distillation	—Unverified	0
Model-Architecture Co-Design for High Performance Temporal GNN Inference on FPGA	Mar 10, 2022	Knowledge Distillation	CodeCode Available	0
Prediction-Guided Distillation for Dense Object Detection	Mar 10, 2022	Dense Object DetectionKnowledge Distillation	CodeCode Available	1
Membership Privacy Protection for Image Translation Models via Adversarial Knowledge Distillation	Mar 10, 2022	Image-to-Image TranslationInference Attack	—Unverified	0
Representation Compensation Networks for Continual Semantic Segmentation	Mar 10, 2022	Class Incremental LearningContinual Learning	CodeCode Available	1
Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability	Mar 10, 2022	Knowledge Distillation	CodeCode Available	1
Efficient Sub-structured Knowledge Distillation	Mar 9, 2022	Knowledge DistillationStructured Prediction	CodeCode Available	0
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting	Mar 9, 2022	Knowledge DistillationTrajectory Forecasting	—Unverified	0
PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors	Mar 8, 2022	DemosaickingKnowledge Distillation	CodeCode Available	0
On Generalizing Beyond Domains in Cross-Domain Continual Learning	Mar 8, 2022	Continual LearningKnowledge Distillation	—Unverified	0
Multi-trial Neural Architecture Search with Lottery Tickets	Mar 8, 2022	Knowledge DistillationNeural Architecture Search	—Unverified	0
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation	Mar 8, 2022	Continual LearningKnowledge Distillation	CodeCode Available	1

Show:10 25 50

← PrevPage 115 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified