Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3726–3750 of 4240 papers

Title	Date	Tasks	Status	Hype
Multi-Frame to Single-Frame: Knowledge Distillation for 3D Object Detection	Sep 24, 2020	3D Object DetectionAutonomous Driving	—Unverified	0
Open-set Short Utterance Forensic Speaker Verification using Teacher-Student Network with Explicit Inductive Bias	Sep 21, 2020	Inductive BiasKnowledge Distillation	—Unverified	0
EI-MTD:Moving Target Defense for Edge Intelligence against Adversarial Attacks	Sep 19, 2020	Knowledge DistillationScheduling	—Unverified	0
Weight Distillation: Transferring the Knowledge in Neural Network Parameters	Sep 19, 2020	Knowledge DistillationMachine Translation	—Unverified	0
Introspective Learning by Distilling Knowledge from Online Self-explanation	Sep 19, 2020	Knowledge Distillation	—Unverified	0
Densely Guided Knowledge Distillation using Multiple Teacher Assistants	Sep 18, 2020	Knowledge DistillationModel Compression	CodeCode Available	1
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning	Sep 17, 2020	Edge-computingKnowledge Distillation	—Unverified	0
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks	Sep 17, 2020	Image ClassificationKnowledge Distillation	CodeCode Available	1
S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning	Sep 17, 2020	Knowledge DistillationMetric Learning	CodeCode Available	1
Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP	Sep 16, 2020	Knowledge Distillation	—Unverified	0
Simplified TinyBERT: Knowledge Distillation for Document Retrieval	Sep 16, 2020	Document RankingKnowledge Distillation	CodeCode Available	1
Noisy Self-Knowledge Distillation for Text Summarization	Sep 15, 2020	Knowledge DistillationSelf-Knowledge Distillation	CodeCode Available	1
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition	Sep 15, 2020	Action RecognitionKnowledge Distillation	—Unverified	0
Autoregressive Knowledge Distillation through Imitation Learning	Sep 15, 2020	Imitation LearningKnowledge Distillation	CodeCode Available	0
SSKD: Self-Supervised Knowledge Distillation for Cross Domain Adaptive Person Re-Identification	Sep 13, 2020	ClusteringDomain Adaptive Person Re-Identification	—Unverified	0
BoostingBERT:Integrating Multi-Class Boosting into BERT for NLP Tasks	Sep 13, 2020	Ensemble LearningKnowledge Distillation	—Unverified	0
DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning	Sep 13, 2020	Graph EmbeddingKnowledge Distillation	—Unverified	0
Extending Label Smoothing Regularization with Self-Knowledge Distillation	Sep 11, 2020	Knowledge DistillationSelf-Knowledge Distillation	—Unverified	0
On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective	Sep 9, 2020	Data AugmentationEfficient Neural Network	—Unverified	0
Simulating Unknown Target Models for Query-Efficient Black-box Attacks	Sep 2, 2020	Knowledge DistillationMeta-Learning	CodeCode Available	1
SAIL: Self-Augmented Graph Contrastive Learning	Sep 2, 2020	Contrastive LearningKnowledge Distillation	—Unverified	0
Lifelong Object Detection	Sep 2, 2020	Knowledge DistillationLifelong learning	—Unverified	0
Classification of Diabetic Retinopathy Using Unlabeled Data and Knowledge Distillation	Sep 1, 2020	ClassificationGeneral Classification	—Unverified	0
Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition	Sep 1, 2020	Action RecognitionImage Generation	CodeCode Available	1
Automatic Assignment of Radiology Examination Protocols Using Pre-trained Language Models with Knowledge Distillation	Sep 1, 2020	Data AugmentationKnowledge Distillation	CodeCode Available	0

Show:10 25 50

← PrevPage 150 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified