Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2976–3000 of 4240 papers

Title	Date	Tasks	Status	Hype
Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation	Dec 22, 2021	Knowledge DistillationMachine Translation	—Unverified	0
Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix	Dec 21, 2021	Knowledge Distillation	—Unverified	0
Supervised Graph Contrastive Pretraining for Text Classification	Dec 21, 2021	ClassificationContrastive Learning	—Unverified	0
Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation	Dec 19, 2021	Anomaly DetectionKnowledge Distillation	CodeCode Available	1
Controlling the Quality of Distillation in Response-Based Network Compression	Dec 19, 2021	Knowledge Distillation	—Unverified	0
LegoDNN: Block-grained Scaling of Deep Neural Networks for Mobile Vision	Dec 18, 2021	Knowledge DistillationModel Compression	—Unverified	0
Distill and De-bias: Mitigating Bias in Face Verification using Knowledge Distillation	Dec 17, 2021	AttributeFace Recognition	—Unverified	0
Knowledge Distillation Improves Stability in Retranslation-based Simultaneous Translation	Dec 17, 2021	Knowledge DistillationTranslation	—Unverified	0
Towards Disturbance-Free Visual Mobile Manipulation	Dec 17, 2021	Collision AvoidanceDeep Reinforcement Learning	CodeCode Available	0
Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition	Dec 17, 2021	image-classificationImage Classification	CodeCode Available	1
Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation	Dec 17, 2021	Contrastive LearningKnowledge Distillation	CodeCode Available	1
Distillation of Human-Object Interaction Contexts for Action Recognition	Dec 17, 2021	Action RecognitionGraph Attention	—Unverified	0
Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching	Dec 17, 2021	Knowledge DistillationSemantic Segmentation	—Unverified	0
Amortized Noisy Channel Neural Machine Translation	Dec 16, 2021	Imitation LearningKnowledge Distillation	—Unverified	0
Learning Cross-Lingual IR from an English Retriever	Dec 15, 2021	Cross-Lingual Information RetrievalInformation Retrieval	CodeCode Available	1
On the Use of External Data for Spoken Named Entity Recognition	Dec 14, 2021	Knowledge Distillationnamed-entity-recognition	CodeCode Available	0
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text	Dec 14, 2021	image-classificationImage Classification	—Unverified	0
A Deep Knowledge Distillation framework for EEG assisted enhancement of single-lead ECG based sleep staging	Dec 14, 2021	ECG based Sleep StagingEEG	CodeCode Available	1
Lifelong Unsupervised Domain Adaptive Person Re-identification with Coordinated Anti-forgetting and Adaptation	Dec 13, 2021	Domain Adaptive Person Re-IdentificationKnowledge Distillation	—Unverified	0
Improving Sequential Recommendations via Bidirectional Temporal Data Augmentation with Pre-training	Dec 13, 2021	Data AugmentationKnowledge Distillation	CodeCode Available	0
Up to 100 Faster Data-free Knowledge Distillation	Dec 12, 2021	Data-free Knowledge DistillationKnowledge Distillation	CodeCode Available	1
DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings	Dec 10, 2021	Contrastive LearningKnowledge Distillation	CodeCode Available	1
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation	Dec 10, 2021	Extractive SummarizationKnowledge Distillation	CodeCode Available	0
Mask-invariant Face Recognition through Template-level Knowledge Distillation	Dec 10, 2021	Face RecognitionKnowledge Distillation	CodeCode Available	1
Mutual Adversarial Training: Learning together is better than going alone	Dec 9, 2021	Knowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 120 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified