Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2276–2300 of 4240 papers

Title	Date	Tasks	Status	Hype
CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT	Dec 22, 2022	Knowledge Distillation	—Unverified	0
UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit Representation	Dec 21, 2022	3D ReconstructionIncremental Learning	CodeCode Available	0
RangeAugment: Efficient Online Augmentation with Range Learning	Dec 20, 2022	Knowledge Distillationobject-detection	—Unverified	0
Fine-Grained Distillation for Long Document Retrieval	Dec 20, 2022	Knowledge DistillationRetrieval	—Unverified	0
Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning	Dec 20, 2022	Knowledge DistillationMachine Translation	—Unverified	0
Adam: Dense Retrieval Distillation with Adaptive Dark Examples	Dec 20, 2022	Knowledge DistillationRetrieval	—Unverified	0
Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization	Dec 19, 2022	Domain GeneralizationKnowledge Distillation	—Unverified	0
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation	Dec 19, 2022	Imitation LearningKnowledge Distillation	—Unverified	0
KNIFE: Distilling Reasoning Knowledge From Free-Text Rationales	Dec 19, 2022	Knowledge DistillationLanguage Modelling	—Unverified	0
Learning Object-level Point Augmentor for Semi-supervised 3D Object Detection	Dec 19, 2022	3D Object DetectionKnowledge Distillation	CodeCode Available	1
Continual Knowledge Distillation for Neural Machine Translation	Dec 18, 2022	Knowledge DistillationMachine Translation	CodeCode Available	0
3D Point Cloud Pre-training with Knowledge Distillation from 2D Images	Dec 17, 2022	Concept AlignmentKnowledge Distillation	—Unverified	0
Teaching Small Language Models to Reason	Dec 16, 2022	GSM8KKnowledge Distillation	—Unverified	0
Swing Distillation: A Privacy-Preserving Knowledge Distillation Framework	Dec 16, 2022	Knowledge DistillationModel Compression	—Unverified	0
Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?	Dec 16, 2022	3D Point Cloud ClassificationFew-Shot 3D Point Cloud Classification	CodeCode Available	1
Gradient-based Intra-attention Pruning on Pre-trained Language Models	Dec 15, 2022	Knowledge Distillation	CodeCode Available	1
Hybrid Paradigm-based Brain-Computer Interface for Robotic Arm Control	Dec 14, 2022	Brain Computer InterfaceEEG	—Unverified	0
Domain Adaptation for Dense Retrieval through Self-Supervision by Pseudo-Relevance Labeling	Dec 13, 2022	Domain AdaptationInformation Retrieval	—Unverified	0
Siamese Sleep Transformer For Robust Sleep Stage Scoring With Self-knowledge Distillation and Selective Batch Sampling	Dec 12, 2022	Knowledge DistillationSelf-Knowledge Distillation	—Unverified	0
Multimodal Matching-aware Co-attention Networks with Mutual Knowledge Distillation for Fake News Detection	Dec 12, 2022	Fake News DetectionImage-text matching	—Unverified	0
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization	Dec 12, 2022	Knowledge DistillationNatural Language Understanding	—Unverified	0
Towards Practical Plug-and-Play Diffusion Models	Dec 12, 2022	Depth EstimationImage Generation	CodeCode Available	1
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging	Dec 12, 2022	Knowledge DistillationQuestion Answering	—Unverified	0
Multi-adversarial Faster-RCNN with Paradigm Teacher for Unrestricted Object Detection	Dec 11, 2022	Domain AdaptationKnowledge Distillation	—Unverified	0
Teaching What You Should Teach: A Data-Based Distillation Method	Dec 11, 2022	Data AugmentationKnowledge Distillation	—Unverified	0

Show:10 25 50

← PrevPage 92 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified