SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3851–3900 of 4240 papers

TitleStatusHype
End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020—0
Channel Distillation: Channel-Wise Attention for Knowledge DistillationCode1
Apprentissage automatique de repr\'esentation de voix \`a l'aide d'une distillation de la connaissance pour le casting vocal (Learning voice representation using knowledge distillation for automatic voice casting )—0
Online Knowledge Distillation via Collaborative LearningCode1
ADINet: Attribute Driven Incremental Network for Retinal Image Classification—0
Distilling Image Dehazing With Heterogeneous Task ImitationCode0
Distilling Cross-Task Knowledge via Relationship MatchingCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Transferring Inductive Biases through Knowledge DistillationCode1
Weight Squeezing: Reparameterization for Compression and Fast Inference—0
Sub-Band Knowledge Distillation Framework for Speech Enhancement—0
Syntactic Structure Distillation Pretraining For Bidirectional Encoders—0
Why distillation helps: a statistical perspective—0
Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech RecognitionCode1
Learning from a Lightweight Teacher for Efficient Knowledge Distillation—0
MicroNet for Efficient Language ModelingCode1
Joint Progressive Knowledge Distillation and Unsupervised Domain AdaptationCode0
Incremental Learning for End-to-End Automatic Speech Recognition—0
Data-Free Network Quantization With Adversarial Knowledge DistillationCode1
Distilling Knowledge from Pre-trained Language Models via Text Smoothing—0
ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural NetworksCode1
MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient EstimationCode1
Heterogeneous Knowledge Distillation using Information Flow ModelingCode1
Improving Non-autoregressive Neural Machine Translation with Monolingual Data—0
Distilling Spikes: Knowledge Distillation in Spiking Neural Networks—0
Language Model Prior for Low-Resource Neural Machine TranslationCode1
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference—0
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning—0
A Tailored Pre-Training Model for Task-Oriented Dialog GenerationCode0
Distilling Knowledge from Refinement in Multiple Instance Detection NetworksCode1
A Study of Non-autoregressive Model for Sequence Generation—0
Making Monolingual Sentence Embeddings Multilingual using Knowledge DistillationCode1
Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation—0
Role-Wise Data Augmentation for Knowledge DistillationCode1
Triplet Loss for Knowledge DistillationCode1
Multimodal and multiview distillation for real-time player detection on a football fieldCode1
Knowledge Distillation for Action Anticipation via Label Smoothing—0
Dark Experience for General Continual Learning: a Strong, Simple BaselineCode1
Building a Multi-domain Neural Machine Translation Model using Knowledge Distillation—0
Smart Inference for Multidigit Convolutional Neural Network based Barcode Decoding—0
Towards Robust Classification with Image Quality Assessment—0
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New OutlooksCode2
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models—0
KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflowCode1
Inter-Region Affinity Distillation for Road Marking SegmentationCode1
Knowledge Distillation for Mobile Edge Computation Offloading—0
On the Effect of Dropping Layers of Pre-trained Transformer ModelsCode1
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression—0
Structure-Level Knowledge Distillation For Multilingual Sequence LabelingCode1
Towards Efficient Unconstrained Palmprint Recognition via Deep Distillation HashingCode1
Show:102550
← PrevPage 78 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified