SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 4101–4150 of 4240 papers

TitleStatusHype
FEED: Feature-level Ensemble Effect for knowledge Distillation—0
Towards a better understanding of Vector Quantized Autoencoders—0
Semi-supervised Acoustic Event Detection based on tri-training—0
Segmenting the FutureCode0
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial NetworksCode0
Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System—0
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding—0
Knowledge Distillation via Route Constrained OptimizationCode1
Feature Fusion for Online Mutual Knowledge DistillationCode0
Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation—0
End-to-End Speech Translation with Knowledge Distillation—0
Visual Relationship Detection with Language prior and SoftmaxCode0
Automatic adaptation of object detectors to new domains using self-trainingCode0
Examining the Mapping Functions of Denoising Autoencoders in Singing Voice Separation—0
Unifying Heterogeneous Classifiers with DistillationCode0
Improved training of binary networks for human pose estimation and image recognition—0
Variational Information Distillation for Knowledge Transfer—0
Knowledge Squeezed Adversarial Network Compression—0
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency—0
Relational Knowledge DistillationCode0
Ultrafast Video Attention Prediction with Coupled Knowledge Distillation—0
Knowledge Distillation for Human Action Anticipation—0
Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization—0
Long-Term Vehicle Localization by Recursive Knowledge Distillation—0
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion—0
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning—0
Correlation Congruence for Knowledge DistillationCode0
A Comprehensive Overhaul of Feature DistillationCode0
Making Neural Machine Reading Comprehension Faster—0
Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation—0
Improving Neural Architecture Search Image Classifiers via Ensemble LearningCode0
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness—0
Knowledge Adaptation for Efficient Semantic Segmentation—0
Structured Knowledge Distillation for Dense PredictionCode0
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation—0
SeizureNet: Multi-Spectral Deep Feature Learning for Seizure Type ClassificationCode0
TKD: Temporal Knowledge Distillation for Active Perception—0
Multilingual Neural Machine Translation with Knowledge DistillationCode0
Improved Knowledge Distillation via Teacher AssistantCode0
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression—0
Compressing GANs using Knowledge Distillation—0
Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks—0
Unsupervised Learning of Neural Networks to Explain Neural Networks (extended abstract)—0
Learning Efficient Detector with Semi-supervised Adaptive DistillationCode0
Stealing Neural Networks via Timing Side Channels—0
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation—0
Learning Student Networks via Feature Embedding—0
Spatial Knowledge Distillation to aid Visual Reasoning—0
Optimizing speed/accuracy trade-off for person re-identification via knowledge distillation—0
An Embarrassingly Simple Approach for Knowledge DistillationCode0
Show:102550
← PrevPage 83 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified