SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2501–2550 of 4240 papers

TitleStatusHype
Real-time Spatio-temporal Action Localization via Learning Motion Representation—0
ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation—0
Rebalancing Multi-Label Class-Incremental Learning—0
Recalling The Forgotten Class Memberships: Unlearned Models Can Be Noisy Labelers to Leak Privacy—0
Recent Advances in Direct Speech-to-text Translation—0
Recent Advances of Continual Learning in Computer Vision: An Overview—0
Membership Privacy for Machine Learning Models Through Knowledge Transfer—0
Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning—0
Rectified Decision Trees: Exploring the Landscape of Interpretable and Effective Machine Learning—0
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness—0
Rectifying the Data Bias in Knowledge Distillation—0
Recurrent knowledge distillation—0
Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translation—0
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation—0
Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation—0
Reducing the Teacher-Student Gap via Adaptive Temperatures—0
RefBERT: Compressing BERT by Referencing to Pre-computed Representations—0
Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation—0
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation—0
Region-aware Knowledge Distillation for Efficient Image-to-Image Translation—0
Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates—0
Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition—0
Reinforced Multi-Teacher Selection for Knowledge Distillation—0
Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition—0
Relation Modeling and Distillation for Learning with Noisy Labels—0
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA—0
Remembering Transformer for Continual Learning—0
Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation—0
Remote Sensing Image Classification with Decoupled Knowledge Distillation—0
Removing Rain Streaks via Task Transfer Learning—0
Representation Consolidation from Multiple Expert Teachers—0
Representation Disparity-aware Distillation for 3D Object Detection—0
Representation Transfer by Optimal Transport—0
Research on Multilingual News Clustering Based on Cross-Language Word Embeddings—0
Research on the Online Update Method for Retrieval-Augmented Generation (RAG) Model with Incremental Learning—0
Residual Knowledge Distillation—0
ResKD: Residual-Guided Knowledge Distillation—0
Resolution-Based Distillation for Efficient Histology Image Classification—0
Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework—0
REFT: Resource-Efficient Federated Training Framework for Heterogeneous and Resource-Constrained Environments—0
Respecting Transfer Gap in Knowledge Distillation—0
Response-based Distillation for Incremental Object Detection—0
Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers—0
Rethinking Attention Mechanism in Time Series Classification—0
Rethinking Feature-Based Knowledge Distillation for Face Recognition—0
Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off—0
Rethinking Knowledge Distillation via Cross-Entropy—0
Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective—0
Rethinking Position Bias Modeling with Knowledge Distillation for CTR Prediction—0
Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective—0
Show:102550
← PrevPage 51 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified