Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2901–2925 of 4240 papers

Title	Date	Tasks	Status	Hype
Meta Knowledge Distillation	Feb 16, 2022	Data AugmentationImage Classification	—Unverified	0
Knowledge Distillation with Deep Supervision	Feb 16, 2022	Knowledge DistillationTransfer Learning	CodeCode Available	0
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation	Feb 16, 2022	Grammatical Error CorrectionKnowledge Distillation	—Unverified	0
FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction	Feb 16, 2022	Active LearningKnowledge Distillation	CodeCode Available	1
No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices	Feb 16, 2022	Federated LearningKnowledge Distillation	—Unverified	0
ZeroGen: Efficient Zero-shot Learning via Dataset Generation	Feb 16, 2022	Data-free Knowledge DistillationDataset Generation	CodeCode Available	1
Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Search	Feb 13, 2022	Contrastive LearningKnowledge Distillation	—Unverified	0
AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation	Feb 13, 2022	Deep LearningDiagnostic	—Unverified	0
Tiny Object Tracking: A Large-scale Dataset and A Baseline	Feb 11, 2022	AttributeKnowledge Distillation	CodeCode Available	2
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning	Feb 9, 2022	AllContrastive Learning	—Unverified	0
Point-Level Region Contrast for Object Detection Pre-Training	Feb 9, 2022	Contrastive LearningKnowledge Distillation	CodeCode Available	1
Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation	Feb 8, 2022	DiversityKnowledge Distillation	CodeCode Available	1
Adaptive Mixing of Auxiliary Losses in Supervised Learning	Feb 7, 2022	DenoisingKnowledge Distillation	CodeCode Available	0
Locally Differentially Private Distributed Deep Learning via Knowledge Distillation	Feb 7, 2022	Deep LearningKnowledge Distillation	CodeCode Available	0
Measuring and Reducing Model Update Regression in Structured Prediction for NLP	Feb 7, 2022	Dependency ParsingKnowledge Distillation	—Unverified	0
Cross domain knowledge compression in realtime optical flow prediction on ultrasound sequences	Feb 4, 2022	Knowledge DistillationOptical Flow Estimation	—Unverified	0
Bootstrapped Representation Learning for Skeleton-Based Action Recognition	Feb 4, 2022	Action RecognitionData Augmentation	—Unverified	0
Iterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition	Feb 4, 2022	ClassificationKnowledge Distillation	—Unverified	0
Local Feature Matching with Transformers for low-end devices	Feb 1, 2022	Knowledge Distillation	CodeCode Available	1
Deep-Disaster: Unsupervised Disaster Detection and Localization Using Visual Data	Jan 31, 2022	HumanitarianKnowledge Distillation	CodeCode Available	0
Improving Robustness by Enhancing Weak Subnets	Jan 30, 2022	Adversarial RobustnessData Augmentation	CodeCode Available	0
Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning	Jan 30, 2022	Knowledge DistillationNetwork Pruning	—Unverified	0
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models	Jan 29, 2022	Inductive BiasKnowledge Distillation	—Unverified	0
Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding	Jan 28, 2022	Graph AttentionKnowledge Distillation	CodeCode Available	1
Dynamic Rectification Knowledge Distillation	Jan 27, 2022	Edge-computingKnowledge Distillation	CodeCode Available	0

Show:10 25 50

← PrevPage 117 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified