Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3201–3225 of 4240 papers

Title	Date	Tasks	Status	Hype
Segmentation with mixed supervision: Confidence maximization helps knowledge distillation	Sep 21, 2021	Image SegmentationKnowledge Distillation	CodeCode Available	1
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation	Sep 21, 2021	Knowledge Distillation	—Unverified	0
Knowledge Distillation with Noisy Labels for Natural Language Understanding	Sep 21, 2021	Knowledge DistillationNatural Language Understanding	—Unverified	0
Releasing Graph Neural Networks with Differential Privacy Guarantees	Sep 18, 2021	Knowledge DistillationPrivacy Preserving	CodeCode Available	0
Towards Full Utilization on Mask Task for Distilling PLMs into NMT	Sep 17, 2021	Knowledge DistillationMachine Translation	—Unverified	0
Distilling Linguistic Context for Language Model Compression	Sep 17, 2021	Knowledge DistillationLanguage Modeling	CodeCode Available	1
Label Assignment Distillation for Object Detection	Sep 16, 2021	Knowledge DistillationObject	—Unverified	0
The NiuTrans System for WNGT 2020 Efficiency Task	Sep 16, 2021	DecoderKnowledge Distillation	CodeCode Available	1
The NiuTrans System for the WMT21 Efficiency Task	Sep 16, 2021	GPUKnowledge Distillation	CodeCode Available	1
New Perspective on Progressive GANs Distillation for One-class Novelty Detection	Sep 15, 2021	DecoderGenerative Adversarial Network	—Unverified	0
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation	Sep 15, 2021	Data AugmentationKnowledge Distillation	CodeCode Available	1
Secure Your Ride: Real-time Matching Success Rate Prediction for Passenger-Driver Pairs	Sep 14, 2021	Decision MakingKnowledge Distillation	—Unverified	0
Multi-Scale Aligned Distillation for Low-Resolution Detection	Sep 14, 2021	Knowledge Distillationobject-detection	CodeCode Available	1
Multihop: Leveraging Complex Models to Learn Accurate Simple Models	Sep 14, 2021	Explainable artificial intelligenceKnowledge Distillation	—Unverified	0
A Note on Knowledge Distillation Loss Function for Object Classification	Sep 14, 2021	Knowledge DistillationModel Compression	—Unverified	0
AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate	Sep 14, 2021	DecoderKnowledge Distillation	—Unverified	0
UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation	Sep 13, 2021	Abstractive Text SummarizationDecoder	—Unverified	0
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding	Sep 13, 2021	Adversarial RobustnessAll	CodeCode Available	1
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation	Sep 13, 2021	Knowledge DistillationLanguage Modeling	—Unverified	0
On the Efficiency of Subclass Knowledge Distillation in Classification Tasks	Sep 12, 2021	Binary ClassificationClassification	—Unverified	0
Federated Ensemble Model-based Reinforcement Learning in Edge Computing	Sep 12, 2021	Autonomous Drivingcontinuous-control	—Unverified	0
Learning to Teach with Student Feedback	Sep 10, 2021	Knowledge Distillation	—Unverified	0
Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation	Sep 10, 2021	Knowledge DistillationQuestion Answering	—Unverified	0
LibFewShot: A Comprehensive Library for Few-shot Learning	Sep 10, 2021	Data AugmentationFew-Shot Image Classification	CodeCode Available	2
Dual Correction Strategy for Ranking Distillation in Top-N Recommender System	Sep 8, 2021	Knowledge DistillationRecommendation Systems	CodeCode Available	0

Show:10 25 50

← PrevPage 129 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified