Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1826–1850 of 4240 papers

Title	Date	Tasks	Status	Hype
Accelerating Molecular Graph Neural Networks via Knowledge Distillation	Jun 26, 2023	Data AugmentationKnowledge Distillation	—Unverified	0
Federated Learning on Non-iid Data via Local and Global Distillation	Jun 26, 2023	Federated LearningKnowledge Distillation	—Unverified	0
Cross Architecture Distillation for Face Recognition	Jun 26, 2023	Face RecognitionKnowledge Distillation	—Unverified	0
Feature Adversarial Distillation for Point Cloud Classification	Jun 25, 2023	ClassificationFAD	—Unverified	0
Enhancing Mapless Trajectory Prediction through Knowledge Distillation	Jun 25, 2023	Autonomous DrivingKnowledge Distillation	—Unverified	0
Robust Spatiotemporal Traffic Forecasting with Reinforced Dynamic Adversarial Training	Jun 25, 2023	Adversarial RobustnessKnowledge Distillation	CodeCode Available	1
Temporal Action Proposal Generation With Action Frequency Adaptive Network	Jun 23, 2023	Knowledge DistillationTemporal Action Proposal Generation	CodeCode Available	0
Incorporating Graph Information in Transformer-based AMR Parsing	Jun 23, 2023	Abstract Meaning RepresentationAMR Parsing	CodeCode Available	0
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes	Jun 23, 2023	Arithmetic ReasoningKnowledge Distillation	—Unverified	0
Knowledge Distillation via Token-level Relationship Graph	Jun 20, 2023	Knowledge DistillationTransfer Learning	—Unverified	0
Recent Advances in Direct Speech-to-text Translation	Jun 20, 2023	Data AugmentationDecoder	—Unverified	0
CrossKD: Cross-Head Knowledge Distillation for Object Detection	Jun 20, 2023	Dense Object DetectionKnowledge Distillation	CodeCode Available	1
FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation	Jun 19, 2023	Action RecognitionFederated Learning	—Unverified	0
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation	Jun 19, 2023	Knowledge DistillationRelation	—Unverified	0
Semi-Supervised Learning for Multi-Label Cardiovascular Diseases Prediction:A Multi-Dataset Study	Jun 18, 2023	Data AugmentationDiagnostic	—Unverified	0
Squeezing nnU-Nets with Knowledge Distillation for On-Board Cloud Detection	Jun 16, 2023	Cloud DetectionKnowledge Distillation	—Unverified	0
Knowledge Distillation for Efficient Audio-Visual Video Captioning	Jun 16, 2023	Audio-Visual Video CaptioningCaption Generation	—Unverified	0
MixedTeacher : Knowledge Distillation for fast inference textural anomaly detection	Jun 16, 2023	Anomaly DetectionKnowledge Distillation	CodeCode Available	0
Coaching a Teachable Student	Jun 16, 2023	CARLA longest6Knowledge Distillation	CodeCode Available	1
Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models	Jun 15, 2023	Data AugmentationKnowledge Distillation	CodeCode Available	0
Self-Knowledge Distillation for Surgical Phase Recognition	Jun 15, 2023	DecoderKnowledge Distillation	—Unverified	0
Heterogeneous Continual Learning	Jun 14, 2023	Continual LearningKnowledge Distillation	—Unverified	0
MiniLLM: Knowledge Distillation of Large Language Models	Jun 14, 2023	Instruction FollowingKnowledge Distillation	CodeCode Available	2
BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation	Jun 13, 2023	Knowledge DistillationSegmentation	CodeCode Available	1
Enhanced Multimodal Representation Learning with Cross-modal KD	Jun 13, 2023	Contrastive LearningEmotion Classification	—Unverified	0

Show:10 25 50

← PrevPage 74 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified