SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3651–3700 of 4240 papers

TitleStatusHype
Topic Modeling for Maternal Health Using Reddit—0
Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space—0
Unsupervised Domain Expansion for Visual CategorizationCode0
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study—0
Fixing the Teacher-Student Knowledge Discrepancy in Distillation—0
Knowledge Distillation By Sparse Representation MatchingCode0
Industry Scale Semi-Supervised Learning for Natural Language Understanding—0
Distilling Virtual Examples for Long-tailed RecognitionCode0
KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning—0
Weakly-Supervised Domain Adaptation of Deep Regression Trackers via Reinforced Knowledge Distillation—0
Hands-on Guidance for Distilling Object Detectors—0
Leaning Compact and Representative Features for Cross-Modality Person Re-IdentificationCode0
A Practical Survey on Faster and Lighter Transformers—0
Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data—0
The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures—0
Student Network Learning via Evolutionary Knowledge Distillation—0
Balanced softmax cross-entropy for incremental learning with and without memory—0
Compacting Deep Neural Networks for Internet of Things: Methods and Applications—0
Online Lifelong Generalized Zero-Shot LearningCode0
Variational Knowledge Distillation for Disease Classification in Chest X-Rays—0
Cost-effective Deployment of BERT Models in Serverless Environment—0
Similarity Transfer for Knowledge Distillation—0
Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation—0
Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition—0
Robustly Optimized and Distilled Training for Natural Language Understanding—0
Robust Model Compression Using Deep HypothesesCode0
A New Training Framework for Deep Neural Network—0
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning—0
Deep Neural Network Models Compression—0
Feature-Align Network with Knowledge Distillation for Efficient Denoising—0
Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network—0
Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition—0
PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation—0
Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks—0
Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation—0
Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation—0
Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification—0
Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation—0
End-to-End Automatic Speech Recognition with Deep Mutual Learning—0
CAP-GAN: Towards Adversarial Robustness with Cycle-consistent Attentional Purification—0
Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification—0
Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation—0
Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation—0
Semantically-Conditioned Negative Samples for Efficient Contrastive Learning—0
Learning Student-Friendly Teacher Networks for Knowledge Distillation—0
NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application—0
Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting—0
Evolutionary Generative Adversarial Networks with Crossover Based Knowledge DistillationCode0
ISP Distillation—0
Network-Agnostic Knowledge Transfer for Medical Image Segmentation—0
Show:102550
← PrevPage 74 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified