Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 3426–3450 of 4240 papers

Title	Date	Tasks	Status	Hype
Distilling Audio-Visual Knowledge by Compositional Contrastive Learning	Apr 22, 2021	Audio Taggingaudio-visual learning	CodeCode Available	1
Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition	Apr 22, 2021	Knowledge Distillation	—Unverified	0
Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices	Apr 21, 2021	Face GenerationFace Model	CodeCode Available	1
Brittle Features May Help Anomaly Detection	Apr 21, 2021	Anomaly DetectionKnowledge Distillation	—Unverified	0
Orderly Dual-Teacher Knowledge Distillation for Lightweight Human Pose Estimation	Apr 21, 2021	BinarizationKnowledge Distillation	—Unverified	0
Balanced Knowledge Distillation for Long-tailed Learning	Apr 21, 2021	Knowledge Distillation	CodeCode Available	1
EduPal leaves no professor behind: Supporting faculty via a peer-powered recommender system	Apr 20, 2021	ChatbotKnowledge Distillation	—Unverified	0
Distill on the Go: Online knowledge distillation in self-supervised learning	Apr 20, 2021	Knowledge DistillationSelf-Supervised Learning	CodeCode Available	1
Knowledge Distillation as Semiparametric Inference	Apr 20, 2021	Knowledge DistillationModel Compression	CodeCode Available	0
Compact CNN Structure Learning by Knowledge Distillation	Apr 19, 2021	Knowledge DistillationModel Compression	—Unverified	0
Distilling Knowledge via Knowledge Review	Apr 19, 2021	Instance SegmentationKnowledge Distillation	CodeCode Available	1
On Learning the Geodesic Path for Incremental Learning	Apr 17, 2021	Incremental LearningKnowledge Distillation	CodeCode Available	1
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos	Apr 16, 2021	Activity RecognitionDiversity	CodeCode Available	1
Counter-Interference Adapter for Multilingual Machine Translation	Apr 16, 2021	Knowledge DistillationMachine Translation	CodeCode Available	1
Continual Learning for Fake Audio Detection	Apr 15, 2021	Continual LearningKnowledge Distillation	—Unverified	0
Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding	Apr 15, 2021	intent-classificationIntent Classification	—Unverified	0
Unsupervised Continual Learning Via Pseudo Labels	Apr 14, 2021	ClusteringContinual Learning	—Unverified	0
Annealing Knowledge Distillation	Apr 14, 2021	image-classificationImage Classification	CodeCode Available	0
Sentence Embeddings by Ensemble Distillation	Apr 14, 2021	Knowledge DistillationSemantic Textual Similarity	—Unverified	0
The Curious Case of Hallucinations in Neural Machine Translation	Apr 14, 2021	HallucinationKnowledge Distillation	CodeCode Available	0
RankDistil: Knowledge Distillation for Ranking	Apr 13, 2021	Document RankingKnowledge Distillation	—Unverified	0
Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer	Apr 13, 2021	Domain AdaptationIncremental Learning	CodeCode Available	1
CXR Segmentation by AdaIN-based Domain Adaptation and Knowledge Distillation	Apr 13, 2021	Domain AdaptationKnowledge Distillation	CodeCode Available	0
Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation	Apr 13, 2021	Knowledge DistillationTriplet	—Unverified	0
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation	Apr 13, 2021	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0

Show:10 25 50

← PrevPage 138 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified