SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2551–2600 of 4240 papers

TitleStatusHype
Rethinking the Knowledge Distillation From the Perspective of Model Calibration—0
Rethinking the Upsampling Layer in Hyperspectral Image Super Resolution—0
Retrieve Anything To Augment Large Language Models—0
Revealing the Two Sides of Data Augmentation: An Asymmetric Distillation-based Win-Win Solution for Open-Set Recognition—0
Reverse-engineering recurrent neural network solutions to a hierarchical inference task for mice—0
Reverse Thinking Makes LLMs Stronger Reasoners—0
Review helps learn better: Temporal Supervised Knowledge Distillation—0
Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search—0
Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study—0
Revisiting Graph based Social Recommendation: A Distillation Enhanced Social Graph Network—0
Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)—0
Revisiting Knowledge Distillation for Object Detection—0
Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving—0
Revisiting Self-Distillation—0
Reward-Based 1-bit Compressed Federated Distillation on Blockchain—0
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd—0
Rich Feature Distillation with Feature Affinity Module for Efficient Image Dehazing—0
RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models—0
RL-based Query Rewriting with Distilled LLM for online E-Commerce Systems—0
RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation—0
Robust Active Distillation—0
Robust Distillation for Worst-class Performance—0
Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples—0
RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness—0
Robust feature knowledge distillation for enhanced performance of lightweight crack segmentation models—0
Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss—0
Robustly Optimized and Distilled Training for Natural Language Understanding—0
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning—0
Robustness to distribution shifts of compressed networks for edge devices—0
Robust Overfitting may be mitigated by properly learned smoothening—0
Robust & Precise Knowledge Distillation-based Novel Context-Aware Predictor for Disease Detection in Brain and Gastrointestinal—0
Role of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data—0
RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models—0
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging—0
RTSR: A Real-Time Super-Resolution Model for AV1 Compressed Content—0
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation—0
S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning—0
S2OSC: A Holistic Semi-Supervised Approach for Open Set Classification—0
S2P3: Self-Supervised Polarimetric Pose Prediction—0
Safe Distillation Box—0
SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior—0
SAM-COD: SAM-guided Unified Framework for Weakly-Supervised Camouflaged Object Detection—0
SAM-Guided Masked Token Prediction for 3D Scene Understanding—0
SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation—0
Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation—0
Sampling to Distill: Knowledge Transfer from Open-World Data—0
Samsung R&D Institute Poland submission to WAT 2021 Indic Language Multilingual Task—0
SC2 Benchmark: Supervised Compression for Split Computing—0
Scalable Collaborative Learning via Representation Sharing—0
Scalable Detection of Salient Entities in News Articles—0
Show:102550
← PrevPage 52 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified