SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2576–2600 of 4240 papers

TitleStatusHype
Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss—0
Robustly Optimized and Distilled Training for Natural Language Understanding—0
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning—0
Robustness to distribution shifts of compressed networks for edge devices—0
Robust Overfitting may be mitigated by properly learned smoothening—0
Robust & Precise Knowledge Distillation-based Novel Context-Aware Predictor for Disease Detection in Brain and Gastrointestinal—0
Role of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data—0
RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models—0
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging—0
RTSR: A Real-Time Super-Resolution Model for AV1 Compressed Content—0
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation—0
S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning—0
S2OSC: A Holistic Semi-Supervised Approach for Open Set Classification—0
S2P3: Self-Supervised Polarimetric Pose Prediction—0
Safe Distillation Box—0
SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior—0
SAM-COD: SAM-guided Unified Framework for Weakly-Supervised Camouflaged Object Detection—0
SAM-Guided Masked Token Prediction for 3D Scene Understanding—0
SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation—0
Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation—0
Sampling to Distill: Knowledge Transfer from Open-World Data—0
Samsung R&D Institute Poland submission to WAT 2021 Indic Language Multilingual Task—0
SC2 Benchmark: Supervised Compression for Split Computing—0
Scalable Collaborative Learning via Representation Sharing—0
Scalable Detection of Salient Entities in News Articles—0
Show:102550
← PrevPage 104 of 170Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified