Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2076–2100 of 4240 papers

Title	Date	Tasks	Status	Hype
Understanding the Role of the Projector in Knowledge Distillation	Mar 20, 2023	image-classificationImage Classification	CodeCode Available	1
More From Less: Self-Supervised Knowledge Distillation for Routine Histopathology Data	Mar 19, 2023	Knowledge Distillation	—Unverified	0
AdaptGuard: Defending Against Universal Attacks for Model Adaptation	Mar 19, 2023	Knowledge Distillationmodel	CodeCode Available	1
An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering	Mar 18, 2023	Graph Question AnsweringKnowledge Distillation	CodeCode Available	0
DC-CCL: Device-Cloud Collaborative Controlled Learning for Large Vision Models	Mar 18, 2023	Knowledge Distillation	—Unverified	0
Crowd Counting with Online Knowledge Learning	Mar 18, 2023	Crowd CountingEdge-computing	—Unverified	0
Confidence Attention and Generalization Enhanced Distillation for Continuous Video Domain Adaptation	Mar 18, 2023	Autonomous DrivingDomain Adaptation	—Unverified	0
Whole-slide-imaging Cancer Metastases Detection and Localization with Limited Tumorous Data	Mar 18, 2023	Knowledge DistillationMedical Image Analysis	CodeCode Available	0
Channel-Aware Distillation Transformer for Depth Estimation on Nano Drones	Mar 18, 2023	Autonomous NavigationDepth Estimation	CodeCode Available	1
TeSLA: Test-Time Self-Learning With Automatic Adversarial Augmentation	Mar 17, 2023	Knowledge DistillationSelf-Learning	CodeCode Available	1
Prototype Knowledge Distillation for Medical Segmentation with Missing Modality	Mar 17, 2023	Image SegmentationKnowledge Distillation	CodeCode Available	1
Distill n' Explain: explaining graph neural networks using simple surrogates	Mar 17, 2023	Knowledge Distillation	CodeCode Available	0
Action knowledge for video captioning with graph neural networks	Mar 16, 2023	Action RecognitionGraph Neural Network	CodeCode Available	1
Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models	Mar 16, 2023	CoLACPU	—Unverified	0
Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval	Mar 16, 2023	Image RetrievalKnowledge Distillation	—Unverified	0
Global Knowledge Calibration for Fast Open-Vocabulary Segmentation	Mar 16, 2023	Knowledge DistillationOpen Vocabulary Semantic Segmentation	CodeCode Available	1
Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models	Mar 16, 2023	Knowledge DistillationMRI segmentation	—Unverified	0
DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model	Mar 16, 2023	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement	Mar 15, 2023	Data AugmentationKnowledge Distillation	CodeCode Available	1
Graph-less Collaborative Filtering	Mar 15, 2023	Collaborative FilteringContrastive Learning	CodeCode Available	1
Cross-resolution Face Recognition via Identity-Preserving Network and Knowledge Distillation	Mar 15, 2023	Face RecognitionKnowledge Distillation	—Unverified	0
DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision	Mar 15, 2023	counterfactualFairness	CodeCode Available	1
Knowledge Distillation from Single to Multi Labels: an Empirical Study	Mar 15, 2023	Classificationimage-classification	CodeCode Available	0
Teacher-Student Knowledge Distillation for Radar Perception on Embedded Accelerators	Mar 14, 2023	Knowledge Distillationobject-detection	—Unverified	0
MetaMixer: A Regularization Strategy for Online Knowledge Distillation	Mar 14, 2023	Knowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 84 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified