Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1426–1450 of 4240 papers

Title	Date	Tasks	Status	Hype
X Modality Assisting RGBT Object Tracking	Dec 27, 2023	Knowledge DistillationObject	—Unverified	0
Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning	Dec 27, 2023	Continual Learninggraph construction	CodeCode Available	0
Group Multi-View Transformer for 3D Shape Analysis with Spatial Encoding	Dec 27, 2023	3D Classification3D Shape Recognition	CodeCode Available	0
AdapterDistillation: Non-Destructive Task Composition with Knowledge Distillation	Dec 26, 2023	Knowledge DistillationRetrieval	—Unverified	0
Cloud-Device Collaborative Learning for Multimodal Large Language Models	Dec 26, 2023	Device-Cloud CollaborationKnowledge Distillation	—Unverified	0
Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments	Dec 26, 2023	Knowledge DistillationMathematical Reasoning	—Unverified	0
Revisiting Knowledge Distillation under Distribution Shift	Dec 25, 2023	Data AugmentationDiversity	CodeCode Available	0
Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation	Dec 22, 2023	Bilevel OptimizationClick-Through Rate Prediction	—Unverified	0
Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold	Dec 22, 2023	Density EstimationImage-to-Image Translation	—Unverified	0
TinySAM: Pushing the Envelope for Efficient Segment Anything Model	Dec 21, 2023	Knowledge DistillationQuantization	CodeCode Available	2
How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark	Dec 21, 2023	Knowledge DistillationLanguage Modeling	—Unverified	0
Object Attribute Matters in Visual Question Answering	Dec 20, 2023	AttributeGraph Neural Network	CodeCode Available	0
DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization	Dec 20, 2023	Knowledge DistillationNatural Language Understanding	—Unverified	0
StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation	Dec 20, 2023	Knowledge Distillation	CodeCode Available	0
Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning	Dec 20, 2023	class-incremental learningClass Incremental Learning	CodeCode Available	0
Federated Learning with Extremely Noisy Clients via Negative Distillation	Dec 20, 2023	Federated LearningKnowledge Distillation	CodeCode Available	1
Expediting Contrastive Language-Image Pretraining via Self-distilled Encoders	Dec 19, 2023	Knowledge Distillation	—Unverified	0
Distilling Autoregressive Models to Obtain High-Performance Non-Autoregressive Solvers for Vehicle Routing Problems with Faster Inference Speed	Dec 19, 2023	Knowledge Distillation	CodeCode Available	1
RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation	Dec 19, 2023	Knowledge DistillationPrediction	—Unverified	0
Decoupled Knowledge with Ensemble Learning for Online Distillation	Dec 18, 2023	Ensemble LearningKnowledge Distillation	CodeCode Available	0
Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models	Dec 17, 2023	Image GenerationKnowledge Distillation	CodeCode Available	1
DistilVPR: Cross-Modal Knowledge Distillation for Visual Place Recognition	Dec 17, 2023	Knowledge DistillationVisual Place Recognition	CodeCode Available	1
Mixed Distillation Helps Smaller Language Model Better Reasoning	Dec 17, 2023	Knowledge DistillationLanguage Modeling	—Unverified	0
Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval	Dec 16, 2023	Image RetrievalKnowledge Distillation	CodeCode Available	0
Simple Image-level Classification Improves Open-vocabulary Object Detection	Dec 16, 2023	Knowledge DistillationObject	CodeCode Available	1

Show:10 25 50

← PrevPage 58 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified