SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1951–2000 of 4240 papers

TitleStatusHype
Efficient Speech Command Recognition Leveraging Spiking Neural Network and Curriculum Learning-based Knowledge Distillation—0
Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding—0
Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation—0
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding—0
Batch Selection and Communication for Active Learning with Edge Labeling—0
Active Large Language Model-based Knowledge Distillation for Session-based Recommendation—0
Improving Neural Machine Translation by Denoising Training—0
Improving Neural ODEs via Knowledge Distillation—0
Efficient Point Cloud Classification via Offline Distillation Framework and Negative-Weight Self-Distillation Technique—0
Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation—0
Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discovery—0
Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS—0
DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser—0
Towards Complementary Knowledge Distillation for Efficient Dense Image Prediction—0
ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model—0
Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation—0
Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recognisers—0
Efficient Object Detection in Optical Remote Sensing Imagery via Attention-based Feature Distillation—0
CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation—0
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task—0
A Survey on Model Compression for Large Language Models—0
Improving Task-Agnostic BERT Distillation with Layer Mapping Search—0
KDSM: An uplift modeling framework based on knowledge distillation and sample matching—0
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation—0
KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation—0
Improving the Transferability of Adversarial Examples by Inverse Knowledge Distillation—0
Improving Video Model Transfer With Dynamic Representation Learning—0
Efficient Machine Translation with Model Pruning and Quantization—0
Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation—0
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation—0
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks—0
In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning—0
ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression—0
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation—0
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation—0
Incremental Classifier Learning Based on PEDCC-Loss and Cosine Distance—0
Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning—0
Incremental Knowledge Based Question Answering—0
Incremental Learning for End-to-End Automatic Speech Recognition—0
Direct Distillation between Different Domains—0
Kendall's τ Coefficient for Logits Distillation—0
Knowledge Adaptation for Efficient Semantic Segmentation—0
Efficient Knowledge Distillation via Curriculum Extraction—0
Efficient Knowledge Distillation of SAM for Medical Image Segmentation—0
Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation—0
Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights—0
Incrementer: Transformer for Class-Incremental Semantic Segmentation With Knowledge Distillation Focusing on Old Class—0
DiReDi: Distillation and Reverse Distillation for AIoT Applications—0
Collective Knowledge Graph Completion with Mutual Knowledge Distillation—0
Efficient Intent-Based Filtering for Multi-Party Conversations Using Knowledge Distillation from LLMs—0
Show:102550
← PrevPage 40 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified