Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 4101–4125 of 4240 papers

Title	Date	Tasks	Status
An Adaptive Random Path Selection Approach for Incremental Learning	Jun 3, 2019	Incremental LearningKnowledge Distillation	CodeCode Available
Knowledge Distillation via Instance Relationship Graph	Jun 1, 2019	Knowledge Distillation	CodeCode Available
On Knowledge distillation from complex networks for response prediction	Jun 1, 2019	Knowledge DistillationQuestion Answering	—Unverified
Structured Knowledge Distillation for Semantic Segmentation	Jun 1, 2019	General Classificationimage-classification	CodeCode Available
Online Distilling from Checkpoints for Neural Machine Translation	Jun 1, 2019	Knowledge DistillationMachine Translation	—Unverified
SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models	May 27, 2019	Knowledge Distillation	CodeCode Available
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation	May 26, 2019	Face HallucinationFace Recognition	—Unverified
Network Pruning via Transformable Architecture Search	May 23, 2019	Knowledge DistillationNetwork Pruning	CodeCode Available
Zero-Shot Knowledge Distillation in Deep Networks	May 20, 2019	Knowledge Distillation	CodeCode Available
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation	May 17, 2019	Knowledge Distillation	CodeCode Available
Creating Lightweight Object Detectors with Model Compression for Deployment on Edge Devices	May 6, 2019	Knowledge DistillationModel Compression	—Unverified
FEED: Feature-level Ensemble Effect for knowledge Distillation	May 1, 2019	Knowledge DistillationTransfer Learning	—Unverified
Towards a better understanding of Vector Quantized Autoencoders	May 1, 2019	Knowledge DistillationMachine Translation	—Unverified
Semi-supervised Acoustic Event Detection based on tri-training	Apr 29, 2019	Event DetectionKnowledge Distillation	—Unverified
Segmenting the Future	Apr 24, 2019	Autonomous DrivingDecision Making	CodeCode Available
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks	Apr 23, 2019	Image GenerationKnowledge Distillation	CodeCode Available
Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System	Apr 21, 2019	Knowledge DistillationModel Compression	—Unverified
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding	Apr 20, 2019	Ensemble LearningKnowledge Distillation	—Unverified
Feature Fusion for Online Mutual Knowledge Distillation	Apr 19, 2019	Knowledge Distillation	CodeCode Available
Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation	Apr 17, 2019	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified
End-to-End Speech Translation with Knowledge Distillation	Apr 17, 2019	Knowledge Distillationspeech-recognition	—Unverified
Visual Relationship Detection with Language prior and Softmax	Apr 16, 2019	Knowledge DistillationRelationship Detection	CodeCode Available
Automatic adaptation of object detectors to new domains using self-training	Apr 15, 2019	Domain AdaptationKnowledge Distillation	CodeCode Available
Examining the Mapping Functions of Denoising Autoencoders in Singing Voice Separation	Apr 12, 2019	DecoderDenoising	—Unverified
Unifying Heterogeneous Classifiers with Distillation	Apr 12, 2019	Knowledge Distillation	CodeCode Available

Show:10 25 50

← PrevPage 165 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified