Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3276–3300 of 4240 papers

Title	Date	Tasks	Status	Hype
Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension	Aug 1, 2021	Knowledge DistillationMachine Reading Comprehension	—Unverified	0
Samsung R&D Institute Poland submission to WAT 2021 Indic Language Multilingual Task	Aug 1, 2021	Domain AdaptationKnowledge Distillation	—Unverified	0
POS-Constrained Parallel Decoding for Non-autoregressive Generation	Aug 1, 2021	Knowledge DistillationPOS	CodeCode Available	0
Trigger is Not Sufficient: Exploiting Frame-aware Knowledge for Implicit Event Argument Extraction	Aug 1, 2021	Event Argument ExtractionKnowledge Distillation	—Unverified	0
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation	Aug 1, 2021	Knowledge DistillationLanguage Modeling	—Unverified	0
Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation	Aug 1, 2021	Cross-Domain Text ClassificationDomain Adaptation	—Unverified	0
Pose-Guided Feature Learning with Knowledge Distillation for Occluded Person Re-Identification	Jul 31, 2021	Knowledge DistillationOccluded Person Re-Identification	—Unverified	0
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals	Jul 30, 2021	ClusteringContrastive Learning	CodeCode Available	0
QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning	Jul 29, 2021	Federated LearningKnowledge Distillation	—Unverified	0
Hierarchical Self-supervised Augmented Knowledge Distillation	Jul 29, 2021	Knowledge DistillationRepresentation Learning	CodeCode Available	1
Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation	Jul 29, 2021	Knowledge DistillationMachine Translation	—Unverified	0
In Defense of the Learning Without Forgetting for Task Incremental Learning	Jul 26, 2021	Continual LearningIncremental Learning	—Unverified	0
Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation	Jul 26, 2021	Handwriting RecognitionHTR	—Unverified	0
IE-GAN: An Improved Evolutionary Generative Adversarial Network Using a New Fitness Function and a Generic Crossover Operator	Jul 25, 2021	Evolutionary AlgorithmsGenerative Adversarial Network	CodeCode Available	0
ROD: Reception-aware Online Distillation for Sparse Graphs	Jul 25, 2021	ClusteringGraph Learning	CodeCode Available	0
The USYD-JD Speech Translation System for IWSLT 2021	Jul 24, 2021	Knowledge DistillationNMT	—Unverified	0
Follow Your Path: a Progressive Method for Knowledge Distillation	Jul 20, 2021	Knowledge Distillation	—Unverified	0
Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System	Jul 20, 2021	Decision MakingKnowledge Distillation	—Unverified	0
Double Similarity Distillation for Semantic Image Segmentation	Jul 19, 2021	Image SegmentationKnowledge Distillation	—Unverified	0
Federated Action Recognition on Heterogeneous Embedded Devices	Jul 18, 2021	Action RecognitionFederated Learning	—Unverified	0
Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search	Jul 15, 2021	Knowledge DistillationNeural Architecture Search	—Unverified	0
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task	Jul 12, 2021	DecoderKnowledge Distillation	—Unverified	0
A Flexible Multi-Task Model for BERT Serving	Jul 12, 2021	Knowledge Distillationmodel	CodeCode Available	0
Technical Report of Team GraphMIRAcles in the WikiKG90M-LSC Track of OGB-LSC @ KDD Cup 2021	Jul 12, 2021	Knowledge DistillationKnowledge Graphs	—Unverified	0
Contrast R-CNN for Continual Learning in Object Detection	Jul 11, 2021	Continual Learningimage-classification	—Unverified	0

Show:10 25 50

← PrevPage 132 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified