Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 4151–4175 of 4240 papers

Title	Date	Tasks	Status	Hype
Few Sample Knowledge Distillation for Efficient Network Compression	Dec 5, 2018	Knowledge DistillationNetwork Pruning	CodeCode Available	0
Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling	Dec 3, 2018	Knowledge DistillationMachine Translation	—Unverified	0
Knowledge Distillation with Feature Maps for Image Classification	Dec 3, 2018	ClassificationGeneral Classification	—Unverified	0
Learning to Specialize with Knowledge Distillation for Visual Question Answering	Dec 1, 2018	General ClassificationGeneral Knowledge	—Unverified	0
KDGAN: Knowledge Distillation with Generative Adversarial Networks	Dec 1, 2018	Knowledge DistillationMulti-Label Learning	—Unverified	0
On Compressing U-net Using Knowledge Distillation	Dec 1, 2018	Knowledge Distillation	—Unverified	0
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks	Nov 26, 2018	General Classificationimage-classification	—Unverified	0
Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation	Nov 25, 2018	CPUFace Model	—Unverified	0
Structured Pruning of Neural Networks with Budget-Aware Regularization	Nov 23, 2018	Knowledge Distillation	—Unverified	0
Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks	Nov 21, 2018	Knowledge DistillationModel Compression	—Unverified	0
Factorized Distillation: Training Holistic Person Re-identification Model by Distilling an Ensemble of Partial ReID Models	Nov 20, 2018	Knowledge DistillationPerson Re-Identification	—Unverified	0
Self-Referenced Deep Learning	Nov 19, 2018	Deep LearningKnowledge Distillation	—Unverified	0
Private Model Compression via Knowledge Distillation	Nov 13, 2018	Knowledge Distillationmodel	—Unverified	0
Sequence-Level Knowledge Distillation for Model Compression of Attention-based Sequence-to-Sequence Speech Recognition	Nov 12, 2018	Knowledge DistillationModel Compression	—Unverified	0
Cogni-Net: Cognitive Feature Learning through Deep Visual Perception	Nov 1, 2018	EEGElectroencephalogram (EEG)	CodeCode Available	0
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation	Oct 29, 2018	Dimensionality ReductionKnowledge Distillation	—Unverified	0
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells	Oct 25, 2018	Depth EstimationDepth Prediction	CodeCode Available	1
Block-wise Intermediate Representation Training for Model Compression	Oct 20, 2018	Knowledge Distillationmodel	—Unverified	0
KTAN: Knowledge Transfer Adversarial Network	Oct 18, 2018	image-classificationImage Classification	—Unverified	0
LIT: Block-wise Intermediate Representation Training for Model Compression	Oct 2, 2018	Knowledge DistillationModel Compression	—Unverified	0
Analyzing Knowledge Distillation in Neural Machine Translation	Oct 1, 2018	Knowledge DistillationMachine Translation	—Unverified	0
Knowledge Distillation from Few Samples	Sep 27, 2018	Knowledge Distillation	—Unverified	0
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System	Sep 19, 2018	Knowledge DistillationLearning-To-Rank	CodeCode Available	0
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection	Sep 16, 2018	ClassificationGeneral Classification	CodeCode Available	1
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations	Sep 13, 2018	Depth EstimationGPU	CodeCode Available	0

Show:10 25 50

← PrevPage 167 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified