Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1751–1775 of 4240 papers

Title	Date	Tasks	Status	Hype
Three Factors to Improve Out-of-Distribution Detection	Aug 2, 2023	Contrastive LearningKnowledge Distillation	—Unverified	0
Spatio-Temporal Branching for Motion Prediction using Motion Increments	Aug 2, 2023	Human motion predictionKnowledge Distillation	CodeCode Available	0
Leveraging Expert Models for Training Deep Neural Networks in Scarce Data Domains: Application to Offline Handwritten Signature Verification	Aug 2, 2023	Knowledge Distillation	—Unverified	0
Ada-DQA: Adaptive Diverse Quality-aware Feature Acquisition for Video Quality Assessment	Aug 1, 2023	DiversityKnowledge Distillation	—Unverified	0
Online Prototype Learning for Online Continual Learning	Aug 1, 2023	Continual LearningKnowledge Distillation	CodeCode Available	1
NormKD: Normalized Logits for Knowledge Distillation	Aug 1, 2023	image-classificationImage Classification	CodeCode Available	1
Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?	Jul 31, 2023	Contrastive LearningKnowledge Distillation	CodeCode Available	0
Federated Learning for Data and Model Heterogeneity in Medical Imaging	Jul 31, 2023	Federated LearningKnowledge Distillation	—Unverified	0
BearingPGA-Net: A Lightweight and Deployable Bearing Fault Diagnosis Network via Decoupled Knowledge Distillation and FPGA Acceleration	Jul 31, 2023	CPUFault Diagnosis	CodeCode Available	1
Sampling to Distill: Knowledge Transfer from Open-World Data	Jul 31, 2023	Data-free Knowledge DistillationKnowledge Distillation	—Unverified	0
Subspace Distillation for Continual Learning	Jul 31, 2023	Continual LearningKnowledge Distillation	CodeCode Available	0
Effective Whole-body Pose Estimation with Two-stages Distillation	Jul 29, 2023	2D Human Pose EstimationKnowledge Distillation	CodeCode Available	4
UPFL: Unsupervised Personalized Federated Learning towards New Clients	Jul 29, 2023	Federated LearningKnowledge Distillation	CodeCode Available	0
f-Divergence Minimization for Sequence-Level Knowledge Distillation	Jul 27, 2023	Knowledge Distillation	CodeCode Available	1
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs	Jul 27, 2023	Document ClassificationKnowledge Distillation	—Unverified	0
Fitting Auditory Filterbanks with Multiresolution Neural Networks	Jul 25, 2023	Inductive BiasKnowledge Distillation	CodeCode Available	1
Mitigating Cross-client GANs-based Attack in Federated Learning	Jul 25, 2023	Data-free Knowledge DistillationFederated Learning	—Unverified	0
MetricGAN-OKD: Multi-Metric Optimization of MetricGAN via Online Knowledge Distillation for Speech Enhancement	Jul 24, 2023	Knowledge DistillationSpeech Enhancement	CodeCode Available	1
A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation	Jul 24, 2023	Knowledge DistillationSemantic Segmentation	—Unverified	0
HeteFedRec: Federated Recommender Systems with Model Heterogeneity	Jul 24, 2023	Knowledge Distillationmodel	—Unverified	0
CLIP-KD: An Empirical Study of CLIP Model Distillation	Jul 24, 2023	Contrastive LearningCross-Modal Retrieval	CodeCode Available	1
Model Compression Methods for YOLOv5: A Review	Jul 21, 2023	Knowledge Distillationmodel	—Unverified	0
DPM-OT: A New Diffusion Probabilistic Model Based on Optimal Transport	Jul 21, 2023	DenoisingKnowledge Distillation	CodeCode Available	1
Distribution Shift Matters for Knowledge Distillation with Webly Collected Images	Jul 21, 2023	Contrastive LearningData-free Knowledge Distillation	—Unverified	0
Quantized Feature Distillation for Network Quantization	Jul 20, 2023	image-classificationImage Classification	—Unverified	0

Show:10 25 50

← PrevPage 71 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified