SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2901–2950 of 4240 papers

TitleStatusHype
SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLPCode0
Distilling Object Detectors With Global KnowledgeCode0
Federated Learning with Privacy-Preserving Ensemble Attention Distillation—0
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging—0
Improving generalizability of distilled self-supervised speech processing models under distorted settingsCode0
Knowledge Distillation approach towards Melanoma DetectionCode0
You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models—0
Probabilistic Integration of Object Level Annotations in Chest X-ray Classification—0
Boosting Graph Neural Networks via Adaptive Knowledge Distillation—0
Integrating Translation Memories into Non-Autoregressive Machine TranslationCode0
SaiT: Sparse Vision Transformers through Adaptive Token PruningCode0
Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR—0
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes—0
Detect, Distill and Update: Learned DB Systems Facing Out of Distribution DataCode0
Linkless Link Prediction via Relational Distillation—0
PP-StructureV2: A Stronger Document Analysis System—0
Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again—0
Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks—0
Students taught by multimodal teachers are superior action recognizers—0
Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization—0
Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation—0
Meta-Ensemble Parameter Learning—0
A Study on the Efficiency and Generalization of Light Hybrid Retrievers—0
Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning—0
Positive Pair Distillation Considered Harmful: Continual Meta Metric Learning for Lifelong Object Re-IdentificationCode0
Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Search—0
Robust Active Distillation—0
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment ClassificationCode0
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation—0
Sentiment Interpretable Logic Tensor Network for Aspect-Term Sentiment Analysis—0
Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model CompressionCode0
Knowledge Transfer with Visual Prompt in multi-modal Dialogue Understanding and Generation—0
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing—0
Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition—0
TAKE: Topic-shift Aware Knowledge sElection for Dialogue GenerationCode0
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language ModelsCode0
Towards a Unified View of Affinity-Based Knowledge Distillation—0
Slimmable Networks for Contrastive Self-supervised LearningCode0
Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation—0
Using Knowledge Distillation to improve interpretable models in a retail banking context—0
Label driven Knowledge Distillation for Federated Learning with non-IID Data—0
Towards Explaining Autonomy with Verbalised Decision Tree States—0
PROD: Progressive Distillation for Dense Retrieval—0
Knowledge Distillation to Ensemble Global and Interpretable Prototype-Based Mammogram Classification Models—0
Joint Speech Activity and Overlap Detection with Multi-Exit Architecture—0
DRKF: Distilled Rotated Kernel Fusion for Efficient Rotation Invariant Descriptors in Local Feature Matching—0
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation—0
Exploring Inconsistent Knowledge Distillation for Object Detection with Data AugmentationCode0
Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition—0
Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation—0
Show:102550
← PrevPage 59 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified