SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 4151–4200 of 4240 papers

TitleStatusHype
Few Sample Knowledge Distillation for Efficient Network CompressionCode0
Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling—0
Knowledge Distillation with Feature Maps for Image Classification—0
Learning to Specialize with Knowledge Distillation for Visual Question Answering—0
KDGAN: Knowledge Distillation with Generative Adversarial Networks—0
On Compressing U-net Using Knowledge Distillation—0
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks—0
Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation—0
Structured Pruning of Neural Networks with Budget-Aware Regularization—0
Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks—0
Factorized Distillation: Training Holistic Person Re-identification Model by Distilling an Ensemble of Partial ReID Models—0
Self-Referenced Deep Learning—0
Private Model Compression via Knowledge Distillation—0
Sequence-Level Knowledge Distillation for Model Compression of Attention-based Sequence-to-Sequence Speech Recognition—0
Cogni-Net: Cognitive Feature Learning through Deep Visual PerceptionCode0
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation—0
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary CellsCode1
Block-wise Intermediate Representation Training for Model Compression—0
KTAN: Knowledge Transfer Adversarial Network—0
LIT: Block-wise Intermediate Representation Training for Model Compression—0
Analyzing Knowledge Distillation in Neural Machine Translation—0
Knowledge Distillation from Few Samples—0
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender SystemCode0
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised DetectionCode1
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric AnnotationsCode0
MotherNets: Rapid Deep Ensemble Learning—0
Label Denoising with Large Ensembles of Heterogeneous Neural Networks—0
RDPD: Rich Data Helps Poor Data via ImitationCode0
Lifelong Learning via Progressive Distillation and Retrospection—0
Attention-Guided Answer Distillation for Machine Reading Comprehension—0
Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks—0
Cooperative Denoising for Distantly Supervised Relation Extraction—0
SlimNets: An Exploration of Deep Model Compression and AccelerationCode0
Self-supervised Knowledge Distillation Using Singular Value DecompositionCode0
Revisiting Distillation and Incremental Classifier LearningCode0
Distillation Techniques for Pseudo-rehearsal Based Incremental LearningCode0
Gradient Adversarial Training of Neural Networks—0
Knowledge Distillation by On-the-Fly Native EnsembleCode0
Coupled End-to-End Transfer Learning With Generalized Fisher Information—0
Collaborative Learning for Deep Neural Networks—0
Channel Gating Neural NetworksCode1
A novel channel pruning method for deep neural network compression—0
Theory and Experiments on Vector Quantized AutoencodersCode0
Visual Relationship Detection Based on Guided Proposals and Semantic Knowledge Distillation—0
Recurrent knowledge distillation—0
Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students—0
Knowledge Distillation with Adversarial Samples Supporting Decision BoundaryCode0
Born Again Neural NetworksCode0
Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation SystemsCode0
Neural Compatibility Modeling with Attentive Knowledge Distillation—0
Show:102550
← PrevPage 84 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified