SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3651–3700 of 4240 papers

TitleStatusHype
Improved training of binary networks for human pose estimation and image recognition—0
Improve Knowledge Distillation via Label Revision and Data Selection—0
Improving Acoustic Scene Classification in Low-Resource Conditions—0
Improving Apple Object Detection with Occlusion-Enhanced Distillation—0
Improving Autoregressive NMT with Non-Autoregressive Model—0
Improving CLIP Robustness with Knowledge Distillation and Self-Training—0
Improving Cone-Beam CT Image Quality with Knowledge Distillation-Enhanced Diffusion Model in Imbalanced Data Settings—0
Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment—0
Improving Defensive Distillation using Teacher Assistant—0
Improving De-Raining Generalization via Neural Reorganization—0
Improving Facial Landmark Detection Accuracy and Efficiency with Knowledge Distillation—0
Improving Feature Generalizability with Multitask Learning in Class Incremental Learning—0
Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition—0
Noise as a Resource for Learning in Knowledge Distillation—0
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging—0
Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning—0
Improving Knowledge Distillation in Transfer Learning with Layer-wise Learning Rates—0
Improving Knowledge Distillation with Teacher's Explanation—0
Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding—0
Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation—0
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding—0
Improving Neural Machine Translation by Denoising Training—0
Improving Neural ODEs via Knowledge Distillation—0
Improving Non-autoregressive Neural Machine Translation with Monolingual Data—0
Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS—0
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation—0
Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation—0
Improving SAM for Camouflaged Object Detection via Dual Stream Adapters—0
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task—0
Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies—0
Improving Task-Agnostic BERT Distillation with Layer Mapping Search—0
Improving Text-based Early Prediction by Distillation from Privileged Time-Series Text—0
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation—0
Improving the Transferability of Adversarial Examples by Inverse Knowledge Distillation—0
Improving Video Model Transfer With Dynamic Representation Learning—0
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation—0
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval—0
In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning—0
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation—0
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement—0
Incremental Classifier Learning Based on PEDCC-Loss and Cosine Distance—0
Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning—0
Incremental Knowledge Based Question Answering—0
Incremental Learning for End-to-End Automatic Speech Recognition—0
Incremental Learning of Acoustic Scenes and Sound Events—0
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs—0
Incrementer: Transformer for Class-Incremental Semantic Segmentation With Knowledge Distillation Focusing on Old Class—0
In Defense of the Learning Without Forgetting for Task Incremental Learning—0
INDUS: Effective and Efficient Language Models for Scientific Applications—0
Industry Scale Semi-Supervised Learning for Natural Language Understanding—0
Show:102550
← PrevPage 74 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified