SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2501–2550 of 4240 papers

TitleStatusHype
Towards Explaining Autonomy with Verbalised Decision Tree States—0
Efficient On-Device Session-Based RecommendationCode1
PROD: Progressive Distillation for Dense Retrieval—0
Rethinking Resolution in the Context of Efficient Video RecognitionCode1
Knowledge Distillation to Ensemble Global and Interpretable Prototype-Based Mammogram Classification Models—0
Joint Speech Activity and Overlap Detection with Multi-Exit Architecture—0
DRKF: Distilled Rotated Kernel Fusion for Efficient Rotation Invariant Descriptors in Local Feature Matching—0
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation—0
Exploring Inconsistent Knowledge Distillation for Object Detection with Data AugmentationCode0
Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition—0
Continual Learning for Class- and Domain-Incremental Semantic Segmentation—0
Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak LabelsCode1
Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation—0
Deliberated Domain Bridging for Domain Adaptive Semantic SegmentationCode1
CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation—0
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation—0
On-Device Domain GeneralizationCode2
TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation—0
PlaStIL: Plastic and Stable Memory-Free Class-Incremental Learning—0
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity MatchingCode0
Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation—0
Switchable Online Knowledge DistillationCode1
Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation—0
In-situ animal behavior classification using knowledge distillation and fixed-point quantization—0
Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning—0
Exploring Target Representations for Masked AutoencodersCode0
Generative Adversarial Super-Resolution at the Edge with Knowledge DistillationCode1
On the Effectiveness of Compact Biomedical TransformersCode1
ViTKD: Practical Guidelines for ViT feature knowledge distillation—0
Domain Generalization for Prostate Segmentation in Transrectal Ultrasound Images: A Multi-center StudyCode1
A Novel Self-Knowledge Distillation Approach with Siamese Representation Learning for Action Recognition—0
Knowledge Distillation for Sustainable Neural Machine Translation—0
A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect DetectionCode1
Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization—0
Membership Inference Attacks by Exploiting Loss TrajectoryCode1
FAKD: Feature Augmented Knowledge Distillation for Semantic SegmentationCode0
Progressive Self-Distillation for Ground-to-Aerial Perception Knowledge TransferCode1
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning StrategyCode0
Goal-Conditioned Q-Learning as Knowledge DistillationCode0
Removing Rain Streaks via Task Transfer Learning—0
Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal PerspectiveCode1
Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge TransferCode0
Dense Depth Distillation with Out-of-Distribution Simulated Images—0
CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual DistillationCode1
Masked Autoencoders Enable Efficient Knowledge DistillersCode1
Debias the Black-box: A Fair Ranking Framework via Knowledge Distillation—0
Lifelong Learning for Neural powered Mixed Integer Programming—0
Semi-supervised Semantic Segmentation with Mutual Knowledge DistillationCode1
FS-BAN: Born-Again Networks for Domain Generalization Few-Shot ClassificationCode0
PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model AdaptationCode1
Show:102550
← PrevPage 51 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified