Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3701–3725 of 4240 papers

Title	Date	Tasks	Status	Hype
Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer	Oct 9, 2020	Decoderimage-classification	—Unverified	0
DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling	Oct 7, 2020	Knowledge DistillationQuestion Answering	—Unverified	0
Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models	Oct 7, 2020	AllKnowledge Distillation	—Unverified	0
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation	Oct 6, 2020	Knowledge DistillationPassage Ranking	CodeCode Available	1
Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review	Oct 6, 2020	ArticlesDeep Learning	—Unverified	0
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers	Oct 6, 2020	Knowledge DistillationMachine Translation	CodeCode Available	0
A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions	Oct 5, 2020	Knowledge DistillationMiscellaneous	—Unverified	0
Improving Neural Topic Models using Knowledge Distillation	Oct 5, 2020	Knowledge DistillationTopic Models	CodeCode Available	1
Self-training Improves Pre-training for Natural Language Understanding	Oct 5, 2020	Data AugmentationFew-Shot Learning	CodeCode Available	1
Lifelong Language Knowledge Distillation	Oct 5, 2020	Knowledge DistillationLanguage Modelling	CodeCode Available	1
Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation	Oct 4, 2020	Cardiac SegmentationImage Segmentation	—Unverified	0
Neighbourhood Distillation: On the benefits of non end-to-end distillation	Oct 2, 2020	Knowledge DistillationNeural Architecture Search	—Unverified	0
Online Knowledge Distillation via Multi-branch Diversity Enhancement	Oct 2, 2020	Diversityimage-classification	—Unverified	0
WeChat Neural Machine Translation Systems for WMT20	Oct 1, 2020	Knowledge DistillationMachine Translation	—Unverified	0
Improved Knowledge Distillation via Full Kernel Matrix Transfer	Sep 30, 2020	Knowledge DistillationModel Compression	CodeCode Available	0
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks	Sep 30, 2020	image-classificationImage Classification	—Unverified	0
Pea-KD: Parameter-efficient and Accurate Knowledge Distillation on BERT	Sep 30, 2020	Knowledge DistillationModel Compression	—Unverified	0
TinyGAN: Distilling BigGAN for Conditional Image Generation	Sep 29, 2020	Conditional Image GenerationImage Generation	CodeCode Available	1
Contrastive Distillation on Intermediate Representations for Language Model Compression	Sep 29, 2020	Knowledge DistillationLanguage Modeling	CodeCode Available	1
Pea-KD: Parameter-efficient and accurate Knowledge Distillation	Sep 28, 2020	Knowledge DistillationModel Compression	—Unverified	0
Kernel Based Progressive Distillation for Adder Neural Networks	Sep 28, 2020	Knowledge Distillation	—Unverified	0
Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach	Sep 28, 2020	Knowledge DistillationLanguage Modelling	—Unverified	0
TernaryBERT: Distillation-aware Ultra-low Bit BERT	Sep 27, 2020	Knowledge DistillationQuantization	CodeCode Available	0
N-LTP: An Open-source Neural Language Technology Platform for Chinese	Sep 24, 2020	Chinese Word SegmentationDependency Parsing	CodeCode Available	3
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey	Sep 24, 2020	Deep Reinforcement LearningDomain Adaptation	—Unverified	0

Show:10 25 50

← PrevPage 149 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified