Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3301–3325 of 4240 papers

Title	Date	Tasks	Status	Hype
Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations	Jul 10, 2021	Facial Expression RecognitionFacial Expression Recognition (FER)	CodeCode Available	1
Lifelong Twin Generative Adversarial Networks	Jul 9, 2021	Knowledge Distillation	—Unverified	0
WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations	Jul 7, 2021	Knowledge DistillationModel Compression	—Unverified	0
Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification	Jul 7, 2021	Classificationimage-classification	CodeCode Available	1
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation	Jul 7, 2021	Fine-Grained Visual RecognitionKnowledge Distillation	—Unverified	0
Confidence Conditioned Knowledge Distillation	Jul 6, 2021	Knowledge Distillation	—Unverified	0
A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation	Jul 6, 2021	Activity RecognitionClassification	—Unverified	0
Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation	Jul 6, 2021	Domain Generalizationimage-classification	—Unverified	0
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer	Jul 6, 2021	Image RetrievalKnowledge Distillation	CodeCode Available	1
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation	Jul 6, 2021	Continual LearningDomain Adaptation	CodeCode Available	0
On The Distribution of Penultimate Activations of Classification Networks	Jul 5, 2021	ClassificationConditional Image Generation	—Unverified	0
Continual Contrastive Learning for Image Classification	Jul 5, 2021	ClassificationContinual Learning	CodeCode Available	0
Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model	Jul 4, 2021	Knowledge DistillationMachine Reading Comprehension	—Unverified	0
Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network	Jul 3, 2021	class-incremental learningClass Incremental Learning	CodeCode Available	1
Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks	Jul 3, 2021	Knowledge DistillationModel Compression	CodeCode Available	0
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation	Jul 3, 2021	Knowledge DistillationModel Compression	CodeCode Available	1
Isotonic Data Augmentation for Knowledge Distillation	Jul 3, 2021	AttributeData Augmentation	—Unverified	0
ESPnet-ST IWSLT 2021 Offline Speech Translation System	Jul 1, 2021	DecoderKnowledge Distillation	—Unverified	0
Revisiting Knowledge Distillation: An Inheritance and Exploration Framework	Jul 1, 2021	Knowledge Distillation	CodeCode Available	0
Knowledge Distillation for Quality Estimation	Jul 1, 2021	Data AugmentationKnowledge Distillation	CodeCode Available	0
Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data	Jun 30, 2021	Federated LearningKnowledge Distillation	—Unverified	0
Learning without Forgetting for 3D Point Cloud Objects	Jun 27, 2021	Knowledge Distillation	CodeCode Available	0
Reward-Based 1-bit Compressed Federated Distillation on Blockchain	Jun 27, 2021	Federated LearningKnowledge Distillation	—Unverified	0
PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation	Jun 25, 2021	Keyword SpottingKnowledge Distillation	—Unverified	0
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains	Jun 25, 2021	Knowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 133 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified