SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3751–3800 of 4240 papers

TitleStatusHype
Large-Scale Generative Data-Free Distillation—0
On Knowledge Distillation for Direct Speech Translation—0
Model Compression Using Optimal Transport—0
Parallel Blockwise Knowledge Distillation for Deep Neural Network CompressionCode0
Reciprocal Supervised Learning Improves Neural Machine TranslationCode0
Multi-head Knowledge Distillation for Model Compression—0
Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains—0
Self-Supervised Generative Adversarial Compression—0
Solvable Model for Inheriting the Regularization through Knowledge Distillation—0
Query Distillation: BERT-based Distillation for Ensemble Ranking—0
Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Evolvability—0
Reverse-engineering recurrent neural network solutions to a hierarchical inference task for mice—0
A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models—0
Real-time Spatio-temporal Action Localization via Learning Motion Representation—0
Adaptive Multiplane Image Generation from a Single Internet Picture—0
torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation—0
Generative Adversarial Simulator—0
MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing—0
A Knowledge Distillation Ensemble Framework for Predicting Short and Long-term Hospitalisation Outcomes from Electronic Health Records DataCode0
Privileged Knowledge Distillation for Online Action Detection—0
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation—0
Generalized Continual Zero-Shot Learning—0
Deep Serial Number: Computational Watermarking for DNN Intellectual Property Protection—0
Digging Deeper into CRNN Model in Chinese Text Images Recognition—0
Online Ensemble Model Compression using Knowledge DistillationCode0
Real-Time Decentralized knowledge Transfer at the EdgeCode0
EGAD: Evolving Graph Representation Learning with Self-Attention and Knowledge Distillation for Live Video Streaming EventsCode0
Distill2Vec: Dynamic Graph Representation Learning with Knowledge DistillationCode0
On Estimating the Training Cost of Conversational Recommendation Systems—0
Knowledge Distillation for Singing Voice DetectionCode0
Ensemble Knowledge Distillation for CTR Prediction—0
Robustness and Diversity Seeking Data-Free Knowledge DistillationCode0
Human-Like Active Learning: Machines Simulating the Human Learning Process—0
Channel Planting for Deep Neural Networks using Knowledge Distillation—0
On Self-Distilling Graph Neural Network—0
Paralinguistic Privacy Protection at the Edge—0
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks—0
Distilling Knowledge by Mimicking FeaturesCode0
Learning to Maximize Speech Quality Directly Using MOS Prediction for Neural Text-to-Speech—0
Data-free Knowledge Distillation for Segmentation using Data-Enriching GANCode0
The NiuTrans Machine Translation Systems for WMT20—0
IIE’s Neural Machine Translation Systems for WMT20—0
HW-TSC’s Participation in the WMT 2020 News Translation Shared Task—0
High Performance Natural Language Processing—0
Using the Past Knowledge to Improve Sentiment Classification—0
Distilling Structured Knowledge for Text-Based Relational Reasoning—0
Fast End-to-end Coreference Resolution for Korean—0
Bridging the Gap between Prior and Posterior Knowledge Selection for Knowledge-Grounded Dialogue Generation—0
FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction—0
MixKD: Towards Efficient Distillation of Large-scale Language Models—0
Show:102550
← PrevPage 76 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified