Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1776–1800 of 4240 papers

Title	Date	Tasks	Status	Hype
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering	Jul 20, 2023	ClusteringData Augmentation	CodeCode Available	0
Reverse Knowledge Distillation: Training a Large Model using a Small One for Retinal Image Matching on Limited Data	Jul 20, 2023	Image RegistrationKeypoint Detection	CodeCode Available	1
LightPath: Lightweight and Scalable Path Representation Learning	Jul 19, 2023	Knowledge DistillationRelational Reasoning	CodeCode Available	0
Teach model to answer questions after comprehending the document	Jul 18, 2023	Knowledge DistillationMachine Reading Comprehension	—Unverified	0
FedDefender: Client-Side Attack-Tolerant Federated Learning	Jul 18, 2023	Federated LearningKnowledge Distillation	CodeCode Available	1
A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future	Jul 18, 2023	Knowledge Distillationobject-detection	CodeCode Available	2
Knowledge Distillation for Object Detection: from generic to remote sensing datasets	Jul 18, 2023	Knowledge DistillationModel Compression	—Unverified	0
Class-relation Knowledge Distillation for Novel Class Discovery	Jul 18, 2023	Knowledge DistillationNovel Class Discovery	CodeCode Available	1
Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain	Jul 17, 2023	Autonomous DrivingKnowledge Distillation	—Unverified	0
DARTS: Double Attention Reference-based Transformer for Super-resolution	Jul 17, 2023	Image Super-ResolutionKnowledge Distillation	CodeCode Available	1
Cumulative Spatial Knowledge Distillation for Vision Transformers	Jul 17, 2023	Inductive BiasKnowledge Distillation	CodeCode Available	1
DOT: A Distillation-Oriented Trainer	Jul 17, 2023	Knowledge Distillation	CodeCode Available	2
Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts	Jul 17, 2023	automatic-speech-translationImitation Learning	CodeCode Available	0
Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages	Jul 16, 2023	Cross-Lingual NERKnowledge Distillation	—Unverified	0
A Survey of Techniques for Optimizing Transformer Inference	Jul 16, 2023	Knowledge DistillationNeural Architecture Search	—Unverified	0
MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning	Jul 16, 2023	Knowledge DistillationMathematical Reasoning	—Unverified	0
Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning	Jul 15, 2023	Contrastive LearningKnowledge Distillation	—Unverified	0
SoccerKDNet: A Knowledge Distillation Framework for Action Recognition in Soccer Videos	Jul 15, 2023	Action RecognitionKnowledge Distillation	—Unverified	0
Learning to Retrieve In-Context Examples for Large Language Models	Jul 14, 2023	In-Context LearningKnowledge Distillation	CodeCode Available	1
DreamTeacher: Pretraining Image Backbones with Deep Generative Models	Jul 14, 2023	Knowledge DistillationRepresentation Learning	—Unverified	0
Multimodal Distillation for Egocentric Action Recognition	Jul 14, 2023	Action RecognitionKnowledge Distillation	CodeCode Available	1
A metric learning approach for endoscopic kidney stone identification	Jul 13, 2023	Few-Shot LearningKnowledge Distillation	—Unverified	0
Frameless Graph Knowledge Distillation	Jul 13, 2023	Graph Representation LearningKnowledge Distillation	CodeCode Available	0
Regression-Oriented Knowledge Distillation for Lightweight Ship Orientation Angle Prediction with Optical Remote Sensing Images	Jul 13, 2023	Knowledge DistillationPrediction	CodeCode Available	0
The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework	Jul 11, 2023	Knowledge DistillationPseudo Label	—Unverified	0

Show:10 25 50

← PrevPage 72 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified