Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2951–2975 of 4240 papers

Title	Date	Tasks	Status	Hype
Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners	Jan 13, 2022	Continual LearningKnowledge Distillation	—Unverified	0
On Exploring Pose Estimation as an Auxiliary Learning Task for Visible-Infrared Person Re-identification	Jan 11, 2022	Auxiliary LearningKnowledge Distillation	CodeCode Available	0
MobileFaceSwap: A Lightweight Framework for Video Face Swapping	Jan 11, 2022	Face SwappingKnowledge Distillation	CodeCode Available	2
FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks	Jan 10, 2022	Data-free Knowledge DistillationFederated Learning	—Unverified	0
Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay	Jan 9, 2022	Data-free Knowledge Distillationimage-classification	CodeCode Available	1
Two-Pass End-to-End ASR Model Compression	Jan 8, 2022	DecoderKnowledge Distillation	—Unverified	0
Microdosing: Knowledge Distillation for GAN based Compression	Jan 7, 2022	Knowledge DistillationVideo Compression	—Unverified	0
Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models	Jan 3, 2022	CPUData Augmentation	—Unverified	0
Class-Incremental Continual Learning into the eXtended DER-verse	Jan 3, 2022	Continual LearningKnowledge Distillation	—Unverified	0
Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation	Jan 1, 2022	Continual LearningContinual Semantic Segmentation	—Unverified	0
Multi-Objective Diverse Human Motion Prediction With Knowledge Distillation	Jan 1, 2022	Autonomous DrivingDiversity	—Unverified	0
Learn From Others and Be Yourself in Heterogeneous Federated Learning	Jan 1, 2022	Continual LearningFederated Learning	CodeCode Available	1
Performance-Aware Mutual Knowledge Distillation for Improving Neural Architecture Search	Jan 1, 2022	Knowledge DistillationNeural Architecture Search	—Unverified	0
Improving Video Model Transfer With Dynamic Representation Learning	Jan 1, 2022	Action ClassificationKnowledge Distillation	—Unverified	0
Distillation Using Oracle Queries for Transformer-Based Human-Object Interaction Detection	Jan 1, 2022	Data AugmentationDecoder	—Unverified	0
Image Restoration using Feature-guidance	Jan 1, 2022	Image RestorationKnowledge Distillation	—Unverified	0
Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data	Jan 1, 2022	Data AugmentationKnowledge Distillation	CodeCode Available	1
Conditional Generative Data-free Knowledge Distillation	Dec 31, 2021	Conditional Image GenerationData-free Knowledge Distillation	—Unverified	0
Data-Free Knowledge Transfer: A Survey	Dec 31, 2021	Data-free Knowledge DistillationDomain Adaptation	—Unverified	0
Confidence-Aware Multi-Teacher Knowledge Distillation	Dec 30, 2021	Knowledge DistillationTransfer Learning	CodeCode Available	1
An Efficient Federated Distillation Learning System for Multi-task Time Series Classification	Dec 30, 2021	Knowledge DistillationTime Series	—Unverified	0
Automatic Mixed-Precision Quantization Search of BERT	Dec 30, 2021	Knowledge DistillationModel Compression	—Unverified	0
Online Adversarial Knowledge Distillation for Graph Neural Networks	Dec 28, 2021	Knowledge Distillation	CodeCode Available	0
Distilling the Knowledge of Romanian BERTs Using Multiple Teachers	Dec 23, 2021	Dialect IdentificationGPU	CodeCode Available	0
Adaptive Beam Search to Enhance On-device Abstractive Summarization	Dec 22, 2021	Abstractive Text SummarizationKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 119 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified