SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3451–3500 of 4240 papers

TitleStatusHype
Class-Balanced Distillation for Long-Tailed Visual RecognitionCode1
Dual Discriminator Adversarial Distillation for Data-free Model Compression—0
Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis—0
Towards Enabling Meta-Learning from Target ModelsCode0
GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent InferenceCode0
Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and RegressionCode0
Content-Aware GAN CompressionCode1
Compressing Visual-linguistic Model via Knowledge Distillation—0
Knowledge Distillation For Wireless Edge LearningCode0
Topic Modeling for Maternal Health Using Reddit—0
Dialect Identification through Adversarial Learning and Knowledge Distillation on Romanian BERT—0
Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space—0
Unsupervised Domain Expansion for Visual CategorizationCode0
Students are the Best Teacher: Exit-Ensemble Distillation with Multi-ExitsCode0
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study—0
Knowledge Distillation By Sparse Representation MatchingCode0
Fixing the Teacher-Student Knowledge Discrepancy in Distillation—0
HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast ImagesCode1
Complementary Relation Contrastive DistillationCode1
Industry Scale Semi-Supervised Learning for Natural Language Understanding—0
Distilling Virtual Examples for Long-tailed RecognitionCode0
Embedding Transfer with Label Relaxation for Improved Metric LearningCode1
KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning—0
Distilling a Powerful Student Model via Online Knowledge DistillationCode1
Multimodal Knowledge ExpansionCode1
A Practical Survey on Faster and Lighter Transformers—0
Distilling Object Detectors via Decoupled FeaturesCode1
Hands-on Guidance for Distilling Object Detectors—0
Leaning Compact and Representative Features for Cross-Modality Person Re-IdentificationCode0
Weakly-Supervised Domain Adaptation of Deep Regression Trackers via Reinforced Knowledge Distillation—0
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine TranslationCode1
Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data—0
The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures—0
Student Network Learning via Evolutionary Knowledge Distillation—0
Balanced softmax cross-entropy for incremental learning with and without memory—0
ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquesCode1
Compacting Deep Neural Networks for Internet of Things: Methods and Applications—0
Variational Knowledge Distillation for Disease Classification in Chest X-Rays—0
Online Lifelong Generalized Zero-Shot LearningCode0
Cost-effective Deployment of BERT Models in Serverless Environment—0
Self-Supervised Adaptation for Video Super-ResolutionCode1
Human-Inspired Multi-Agent Navigation using Knowledge DistillationCode1
Similarity Transfer for Knowledge Distillation—0
Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation—0
Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition—0
Robustly Optimized and Distilled Training for Natural Language Understanding—0
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge DistillationCode1
Robust Model Compression Using Deep HypothesesCode0
A New Training Framework for Deep Neural Network—0
Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve BackbonesCode1
Show:102550
← PrevPage 70 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified