Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2901–2925 of 4240 papers

Title	Date	Tasks	Status
SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP	Oct 18, 2022	Knowledge DistillationNode Classification	CodeCode Available
Distilling Object Detectors With Global Knowledge	Oct 17, 2022	Knowledge DistillationObject	CodeCode Available
Federated Learning with Privacy-Preserving Ensemble Attention Distillation	Oct 16, 2022	Federated Learningimage-classification	—Unverified
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging	Oct 15, 2022	ClassificationKnowledge Distillation	—Unverified
Improving generalizability of distilled self-supervised speech processing models under distorted settings	Oct 14, 2022	Knowledge Distillation	CodeCode Available
Knowledge Distillation approach towards Melanoma Detection	Oct 14, 2022	Knowledge DistillationTAG	CodeCode Available
You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models	Oct 13, 2022	Cross-Lingual TransferKnowledge Distillation	—Unverified
Probabilistic Integration of Object Level Annotations in Chest X-ray Classification	Oct 13, 2022	Knowledge DistillationVariational Inference	—Unverified
Boosting Graph Neural Networks via Adaptive Knowledge Distillation	Oct 12, 2022	Graph ClassificationGraph Mining	—Unverified
Integrating Translation Memories into Non-Autoregressive Machine Translation	Oct 12, 2022	Knowledge DistillationMachine Translation	CodeCode Available
SaiT: Sparse Vision Transformers through Adaptive Token Pruning	Oct 11, 2022	Knowledge Distillation	CodeCode Available
Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR	Oct 11, 2022	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes	Oct 11, 2022	Active LearningKnowledge Distillation	—Unverified
Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data	Oct 11, 2022	Knowledge DistillationSynthetic Data Generation	CodeCode Available
Linkless Link Prediction via Relational Distillation	Oct 11, 2022	Knowledge DistillationLink Prediction	—Unverified
PP-StructureV2: A Stronger Document Analysis System	Oct 11, 2022	Key Information ExtractionKnowledge Distillation	—Unverified
Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again	Oct 10, 2022	Knowledge Distillation	—Unverified
Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks	Oct 10, 2022	domain classificationintent-classification	—Unverified
Students taught by multimodal teachers are superior action recognizers	Oct 9, 2022	Action RecognitionKnowledge Distillation	—Unverified
Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization	Oct 7, 2022	Knowledge Distillationspeaker-diarization	—Unverified
Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation	Oct 5, 2022	Graph Representation LearningKnowledge Distillation	—Unverified
Meta-Ensemble Parameter Learning	Oct 5, 2022	Knowledge DistillationMeta-Learning	—Unverified
A Study on the Efficiency and Generalization of Light Hybrid Retrievers	Oct 4, 2022	Adversarial AttackContrastive Learning	—Unverified
Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning	Oct 4, 2022	Federated LearningKnowledge Distillation	—Unverified
Positive Pair Distillation Considered Harmful: Continual Meta Metric Learning for Lifelong Object Re-Identification	Oct 4, 2022	Knowledge DistillationMetric Learning	CodeCode Available

Show:10 25 50

← PrevPage 117 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified