SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 4201–4240 of 4240 papers

TitleStatusHype
Visual Relationship Detection Based on Guided Proposals and Semantic Knowledge Distillation—0
Recurrent knowledge distillation—0
Knowledge Distillation with Adversarial Samples Supporting Decision BoundaryCode0
Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students—0
Born Again Neural NetworksCode0
Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation SystemsCode0
Neural Compatibility Modeling with Attentive Knowledge Distillation—0
Few-shot learning of neural networks from scratch by pseudo example optimization—0
Model compression for faster structural separation of macromolecules captured by Cellular Electron Cryo-Tomography—0
Faster gaze prediction with dense networks and Fisher pruningCode0
Deep Net Triage: Analyzing the Importance of Network Layers via Structural Compression—0
Generation and Consolidation of Recollections for Efficient Deep Lifelong Learning—0
Learning Deep and Compact Models for Gesture RecognitionCode0
NestedNet: Learning Nested Sparse Structures in Deep Neural Networks—0
StrassenNets: Deep Learning with a Multiplication BudgetCode0
Learning Efficient Object Detection Models with Knowledge Distillation—0
Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN—0
MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face ImagesCode0
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy—0
Non-Autoregressive Neural Machine TranslationCode0
A Survey of Model Compression and Acceleration for Deep Neural Networks—0
Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification—0
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks—0
Knowledge Distillation for Bilingual Dictionary Induction—0
A Joint Sequential and Relational Model for Frame-Semantic Parsing—0
Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation—0
WebChild 2.0 : Fine-Grained Commonsense Knowledge Distillation—0
A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning—0
TIP: Typifying the Interpretability of Procedures—0
Knowledge distillation using unlabeled mismatched images—0
Collaborative Deep Reinforcement LearningCode0
Knowledge Adaptation: Teaching to Adapt—0
Ensemble Distillation for Neural Machine Translation—0
Neural Machine Translation from Simplified Translations—0
In Teacher We Trust: Learning Compressed Models for Pedestrian Detection—0
A scalable convolutional neural network for task-specified scenarios via knowledge distillation—0
Knowledge Distillation for Small-footprint Highway Networks—0
Adapting Models to Signal Degradation using Distillation—0
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain—0
Distilling Model KnowledgeCode0
Show:102550
← PrevPage 85 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified