Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 1451–1475 of 4240 papers

Title	Date	Tasks	Status	Hype
FastSR-NeRF: Improving NeRF Efficiency on Consumer Devices with A Simple Super-Resolution Pipeline	Dec 15, 2023	GPUKnowledge Distillation	—Unverified	0
Student as an Inherent Denoiser of Noisy Teacher	Dec 15, 2023	Knowledge DistillationLanguage Modeling	—Unverified	0
WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge	Dec 15, 2023	Information RetrievalKnowledge Distillation	CodeCode Available	0
MobileSAMv2: Faster Segment Anything to Everything	Dec 15, 2023	DecoderKnowledge Distillation	CodeCode Available	5
Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference	Dec 15, 2023	DecoderDenoising	CodeCode Available	2
RankDVQA-mini: Knowledge Distillation-Driven Deep Video Quality Assessment	Dec 14, 2023	Knowledge DistillationModel Compression	—Unverified	0
RdimKD: Generic Distillation Paradigm by Dimensionality Reduction	Dec 14, 2023	Dimensionality ReductionKnowledge Distillation	—Unverified	0
Generative Model-based Feature Knowledge Distillation for Action Recognition	Dec 14, 2023	Action DetectionAction Recognition	CodeCode Available	1
Unraveling Key Factors of Knowledge Distillation	Dec 14, 2023	Knowledge DistillationMachine Translation	—Unverified	0
Efficient speech detection in environmental audio using acoustic recognition and knowledge distillation	Dec 14, 2023	Knowledge DistillationModel Selection	—Unverified	0
COMBHelper: A Neural Approach to Reduce Search Space for Graph Combinatorial Problems	Dec 14, 2023	Combinatorial OptimizationGraph Neural Network	CodeCode Available	0
SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector	Dec 14, 2023	Knowledge DistillationObject	CodeCode Available	1
CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data	Dec 14, 2023	Contrastive LearningFederated Learning	CodeCode Available	1
Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models	Dec 13, 2023	Image GenerationKnowledge Distillation	—Unverified	0
KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp Segmentation	Dec 13, 2023	Knowledge DistillationMedical Image Segmentation	CodeCode Available	1
Cooperative Learning for Cost-Adaptive Inference	Dec 13, 2023	Knowledge Distillation	—Unverified	0
Mutual-Learning Knowledge Distillation for Nighttime UAV Tracking	Dec 13, 2023	Knowledge Distillation	CodeCode Available	0
Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach	Dec 12, 2023	Knowledge DistillationOffline RL	CodeCode Available	1
A dynamic interactive learning framework for automated 3D medical image segmentation	Dec 11, 2023	Image RegistrationImage Segmentation	—Unverified	0
NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation	Dec 10, 2023	Knowledge Distillation	—Unverified	0
Fake It Till Make It: Federated Learning with Consensus-Oriented Generation	Dec 10, 2023	Federated LearningKnowledge Distillation	—Unverified	0
IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment	Dec 10, 2023	Incremental LearningKnowledge Distillation	—Unverified	0
Understanding the Effect of Model Compression on Social Bias in Large Language Models	Dec 9, 2023	Knowledge DistillationModel Compression	CodeCode Available	0
Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation	Dec 9, 2023	Adversarial RobustnessFairness	CodeCode Available	0
Localized Symbolic Knowledge Distillation for Visual Commonsense Models	Dec 8, 2023	Image DescriptionInstruction Following	CodeCode Available	0

Show:10 25 50

← PrevPage 59 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified