SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2501–2525 of 4240 papers

TitleStatusHype
Real-time Spatio-temporal Action Localization via Learning Motion Representation—0
ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation—0
Rebalancing Multi-Label Class-Incremental Learning—0
Recalling The Forgotten Class Memberships: Unlearned Models Can Be Noisy Labelers to Leak Privacy—0
Recent Advances in Direct Speech-to-text Translation—0
Recent Advances of Continual Learning in Computer Vision: An Overview—0
Membership Privacy for Machine Learning Models Through Knowledge Transfer—0
Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning—0
Rectified Decision Trees: Exploring the Landscape of Interpretable and Effective Machine Learning—0
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness—0
Rectifying the Data Bias in Knowledge Distillation—0
Recurrent knowledge distillation—0
Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translation—0
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation—0
Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation—0
Reducing the Teacher-Student Gap via Adaptive Temperatures—0
RefBERT: Compressing BERT by Referencing to Pre-computed Representations—0
Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation—0
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation—0
Region-aware Knowledge Distillation for Efficient Image-to-Image Translation—0
Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates—0
Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition—0
Reinforced Multi-Teacher Selection for Knowledge Distillation—0
Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition—0
Relation Modeling and Distillation for Learning with Noisy Labels—0
Show:102550
← PrevPage 101 of 170Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified