Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 2601–2625 of 4240 papers

Title	Date	Tasks	Status	Hype
Aware of the History: Trajectory Forecasting with the Local Behavior Data	Jul 20, 2022	Knowledge DistillationPrediction	—Unverified	0
Model Compression for Resource-Constrained Mobile Robots	Jul 20, 2022	Knowledge Distillationmodel	—Unverified	0
Knowledge distillation with a class-aware loss for endoscopic disease detection	Jul 19, 2022	DiagnosticKnowledge Distillation	—Unverified	0
Context Unaware Knowledge Distillation for Image Retrieval	Jul 19, 2022	Image RetrievalKnowledge Distillation	CodeCode Available	0
FedX: Unsupervised Federated Learning with Cross Knowledge Distillation	Jul 19, 2022	Contrastive LearningFederated Learning	CodeCode Available	1
Informative knowledge distillation for image anomaly segmentation	Jul 19, 2022	Anomaly DetectionAnomaly Segmentation	CodeCode Available	1
Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution	Jul 18, 2022	Image Super-ResolutionKnowledge Distillation	—Unverified	0
Class-incremental Novel Class Discovery	Jul 18, 2022	Incremental LearningKnowledge Distillation	CodeCode Available	1
Rethinking Data Augmentation for Robust Visual Question Answering	Jul 18, 2022	Data AugmentationKnowledge Distillation	CodeCode Available	1
TSPipe: Learn from Teacher Faster with Pipelines	Jul 17, 2022	GPUKnowledge Distillation	CodeCode Available	0
Subclass Knowledge Distillation with Known Subclass Labels	Jul 17, 2022	Binary ClassificationKnowledge Distillation	—Unverified	0
SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection	Jul 16, 2022	Anomaly DetectionKnowledge Distillation	—Unverified	0
Multi-Level Branched Regularization for Federated Learning	Jul 14, 2022	Federated LearningKnowledge Distillation	CodeCode Available	1
Dynamic Low-Resolution Distillation for Cost-Efficient End-to-End Text Spotting	Jul 14, 2022	global-optimizationKnowledge Distillation	—Unverified	0
Rethinking Attention Mechanism in Time Series Classification	Jul 14, 2022	ClassificationKnowledge Distillation	—Unverified	0
Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources	Jul 14, 2022	Knowledge Distillation	CodeCode Available	1
Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models	Jul 14, 2022	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	—Unverified	0
Rich Feature Distillation with Feature Affinity Module for Efficient Image Dehazing	Jul 13, 2022	Contrastive Learningimage-classification	—Unverified	0
DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning	Jul 13, 2022	Knowledge DistillationLinear evaluation	—Unverified	0
ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech	Jul 13, 2022	DenoisingGPU	CodeCode Available	3
Re2G: Retrieve, Rerank, Generate	Jul 13, 2022	Fact CheckingFact Verification	CodeCode Available	1
SlimSeg: Slimmable Semantic Segmentation with Boundary Supervision	Jul 13, 2022	Knowledge DistillationSegmentation	—Unverified	0
Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices	Jul 12, 2022	Emotion RecognitionKeyword Spotting	CodeCode Available	0
Contrastive Deep Supervision	Jul 12, 2022	Contrastive LearningFine-Grained Image Classification	CodeCode Available	1
Normalized Feature Distillation for Semantic Segmentation	Jul 12, 2022	Knowledge DistillationModel Compression	—Unverified	0

Show:10 25 50

← PrevPage 105 of 170Next →

All datasets ImageNet CIFAR-100 COCO (Common Objects in Context)COCO 2017 val PASCAL VOC KITTI

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	ScaleKD (T:BEiT-L S:ViT-B/14)	Top-1 accuracy %	86.43	—	Unverified
2	ScaleKD (T:Swin-L S:ViT-B/16)	Top-1 accuracy %	85.53	—	Unverified
3	ScaleKD (T:Swin-L S:ViT-S/16)	Top-1 accuracy %	83.93	—	Unverified
4	ScaleKD (T:Swin-L S:Swin-T)	Top-1 accuracy %	83.8	—	Unverified
5	KD++(T: regnety-16GF S:ViT-B)	Top-1 accuracy %	83.6	—	Unverified
6	VkD (T:RegNety 160 S:DeiT-S)	Top-1 accuracy %	82.9	—	Unverified
7	SpectralKD (T:Swin-S S:Swin-T)	Top-1 accuracy %	82.7	—	Unverified
8	ScaleKD (T:Swin-L S:ResNet-50)	Top-1 accuracy %	82.55	—	Unverified
9	DiffKD (T:Swin-L S: Swin-T)	Top-1 accuracy %	82.5	—	Unverified
10	DIST (T: Swin-L S: Swin-T)	Top-1 accuracy %	82.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SRD (T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	79.86	—	Unverified
2	shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	78.76	—	Unverified
3	MV-MR (T: CLIP/ViT-B-16 S: resnet50)	Top-1 Accuracy (%)	78.6	—	Unverified
4	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	78.28	—	Unverified
5	resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])	Top-1 Accuracy (%)	78.08	—	Unverified
6	ReviewKD++(T:resnet-32x4, S:shufflenet-v2)	Top-1 Accuracy (%)	77.93	—	Unverified
7	ReviewKD++(T:resnet-32x4, S:shufflenet-v1)	Top-1 Accuracy (%)	77.68	—	Unverified
8	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	77.5	—	Unverified
9	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.68	—	Unverified
10	resnet8x4 (T: resnet32x4 S: resnet8x4)	Top-1 Accuracy (%)	76.31	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	77.16	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	73.73	—	Unverified
3	ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small)	box AP	47.6	—	Unverified
4	ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)	mask AP	42.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))	AP@0.5	61.8	—	Unverified
2	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))	AP@0.5	57.96	—	Unverified
3	ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))	AP@0.5	55.18	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	LSHFM (T: ResNet101 S: ResNet50)	mAP	93.17	—	Unverified
2	LSHFM (T: ResNet101 S: MobileNetV2)	mAP	90.14	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	TIE-KD (T: Adabins S: MobileNetV2)	RMSE	2.43	—	Unverified