Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2726–2750 of 4240 papers

Title	Date	Tasks	Status	Hype
Directed Acyclic Transformer for Non-Autoregressive Machine Translation	May 16, 2022	Knowledge DistillationMachine Translation	CodeCode Available	1
Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering	May 15, 2022	Domain GeneralizationKnowledge Distillation	—Unverified	0
Knowledge Distillation Meets Open-Set Semi-Supervised Learning	May 13, 2022	Face RecognitionKnowledge Distillation	CodeCode Available	1
"Teaching Independent Parts Separately" (TIPSy-GAN) : Improving Accuracy and Stability in Unsupervised Adversarial 2D to 3D Pose Estimation	May 12, 2022	3D Human Pose Estimation3D Pose Estimation	—Unverified	0
D3T-GAN: Data-Dependent Domain Transfer GANs for Few-shot Image Generation	May 12, 2022	Image GenerationKnowledge Distillation	—Unverified	0
Knowledge Distillation for Multi-Target Domain Adaptation in Real-Time Person Re-Identification	May 12, 2022	Domain AdaptationKnowledge Distillation	CodeCode Available	0
DistilProtBert: A distilled protein language model used to distinguish between real proteins and their randomly shuffled counterparts	May 10, 2022	Dimensionality ReductionKnowledge Distillation	CodeCode Available	1
Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning	May 9, 2022	Few-Shot Object DetectionKnowledge Distillation	—Unverified	0
Data-Free Adversarial Knowledge Distillation for Graph Neural Networks	May 8, 2022	Generative Adversarial NetworkGraph Classification	—Unverified	0
ConceptDistil: Model-Agnostic Distillation of Concept Explanations	May 7, 2022	Explainable ModelsKnowledge Distillation	—Unverified	0
Automatic Block-wise Pruning with Auxiliary Gating Structures for Deep Convolutional Neural Networks	May 7, 2022	Knowledge DistillationModel Compression	—Unverified	0
Distilling Inter-Class Distance for Semantic Segmentation	May 7, 2022	Knowledge DistillationPosition	—Unverified	0
Collective Relevance Labeling for Passage Retrieval	May 6, 2022	Information RetrievalKnowledge Distillation	CodeCode Available	0
Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks	May 5, 2022	Knowledge Distillation	CodeCode Available	0
Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systems	May 5, 2022	Adversarial AttackKnowledge Distillation	—Unverified	0
A Deep Reinforcement Learning Framework for Rapid Diagnosis of Whole Slide Pathological Images	May 5, 2022	Deep Reinforcement LearningKnowledge Distillation	—Unverified	0
Spot-adaptive Knowledge Distillation	May 5, 2022	Knowledge Distillation	CodeCode Available	1
FedSPLIT: One-Shot Federated Recommendation System Based on Non-negative Joint Matrix Factorization and Knowledge Distillation	May 4, 2022	Collaborative FilteringFederated Learning	—Unverified	0
Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss	May 4, 2022	DescriptiveKnowledge Distillation	—Unverified	0
Knowledge Distillation of Russian Language Models with Reduction of Vocabulary	May 4, 2022	Knowledge Distillation	CodeCode Available	0
Generalized Knowledge Distillation via Relationship Matching	May 4, 2022	Few-Shot LearningIncremental Learning	CodeCode Available	0
Masked Generative Distillation	May 3, 2022	image-classificationImage Classification	CodeCode Available	2
FedDKD: Federated Learning with Decentralized Knowledge Distillation	May 2, 2022	Federated LearningKnowledge Distillation	—Unverified	0
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation	May 1, 2022	Knowledge DistillationTranslation	CodeCode Available	0
Knowledge Distillation Meets Few-Shot Learning: An Approach for Few-Shot Intent Classification Within and Across Domains	May 1, 2022	Cross-Domain Few-Shotcross-domain few-shot learning	—Unverified	0

Show:10 25 50

← PrevPage 110 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified