SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3501–3550 of 4240 papers

TitleStatusHype
Parser-Free Virtual Try-on via Distilling Appearance FlowsCode1
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning—0
Adaptive Multi-Teacher Multi-level Knowledge DistillationCode1
Teachers Do More Than Teach: Compressing Image-to-Image ModelsCode1
Distributed Dynamic Map Fusion via Federated Learning for Intelligent Networked VehiclesCode1
Deep Neural Network Models Compression—0
Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation FrameworkCode1
General Instance Distillation for Object DetectionCode1
Feature-Align Network with Knowledge Distillation for Efficient Denoising—0
Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot LearningCode1
Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network—0
Training Generative Adversarial Networks in One StageCode1
Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition—0
Distilling Knowledge via Intermediate ClassifiersCode1
PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation—0
Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks—0
Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-DistillationCode1
Localization Distillation for Dense Object DetectionCode1
Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation—0
Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation—0
CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray SegmentationCode1
Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification—0
End-to-End Automatic Speech Recognition with Deep Mutual Learning—0
Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation—0
Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation—0
CAP-GAN: Towards Adversarial Robustness with Cycle-consistent Attentional Purification—0
Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification—0
Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation—0
Learning Student-Friendly Teacher Networks for Knowledge Distillation—0
Semantically-Conditioned Negative Samples for Efficient Contrastive Learning—0
NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application—0
Show, Attend and Distill:Knowledge Distillation via Attention-based Feature MatchingCode1
ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning ModelsCode1
Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting—0
Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff PerspectiveCode1
Evolutionary Generative Adversarial Networks with Crossover Based Knowledge DistillationCode0
ISP Distillation—0
Network-Agnostic Knowledge Transfer for Medical Image Segmentation—0
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay BufferCode1
Bridging the gap between Human Action Recognition and Online Action Detection—0
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer—0
Deep Epidemiological Modeling by Black-box Knowledge Distillation: An Accurate Deep Learning Model for COVID-19—0
Learning to Augment for Data-Scarce Domain BERT Knowledge Distillation—0
Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains—0
Incremental Knowledge Based Question Answering—0
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization—0
Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Data—0
SEED: Self-supervised Distillation For Visual RepresentationCode1
Interpretable discovery of new semiconductors with machine learning—0
Resolution-Based Distillation for Efficient Histology Image Classification—0
Show:102550
← PrevPage 71 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified