SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1451–1500 of 4240 papers

TitleStatusHype
Enhancing Scalability in Recommender Systems through Lottery Ticket Hypothesis and Knowledge Distillation-based Neural Network Pruning—0
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition—0
Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentation—0
Enhancing Review Comprehension with Domain-Specific Commonsense—0
Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits—0
Expediting Contrastive Language-Image Pretraining via Self-distilled Encoders—0
Experimentation in Content Moderation using RWKV—0
Experimenting with Knowledge Distillation techniques for performing Brain Tumor Segmentation—0
Explainability-Driven Leaf Disease Classification Using Adversarial Training and Knowledge Distillation—0
Explainable Knowledge Distillation for On-device Chest X-Ray Classification—0
Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning—0
Explaining Knowledge Distillation by Quantifying the Knowledge—0
ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval—0
Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation—0
A General Multiple Data Augmentation Based Framework for Training Deep Neural Networks—0
Explicit Connection Distillation—0
A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition—0
Explicit Knowledge Transfer for Weakly-Supervised Code Generation—0
FlyKD: Graph Knowledge Distillation on the Fly with Curriculum Learning—0
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR—0
Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation—0
Enhancing Mapless Trajectory Prediction through Knowledge Distillation—0
Exploring compressibility of transformer based text-to-music (TTM) models—0
Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch—0
Compression of end-to-end non-autoregressive image-to-speech system for low-resourced devices—0
Compression of Deep Learning Models for Text: A Survey—0
Generalized Supervised Contrastive Learning—0
Exploring Extreme Quantization in Spiking Language Models—0
Compression of Acoustic Event Detection Models With Quantized Distillation—0
Continual Learning for Class- and Domain-Incremental Semantic Segmentation—0
FLAR: A Unified Prototype Framework for Few-Sample Lifelong Active Recognition—0
For the Misgendered Chinese in Gender Bias Research: Multi-Task Learning with Knowledge Distillation for Pinyin Name-Gender Prediction—0
Compressing Visual-linguistic Model via Knowledge Distillation—0
Fully Synthetic Data Improves Neural Machine Translation with Knowledge Distillation—0
Enhancing Generalization in Chain of Thought Reasoning for Smaller Models—0
A Theoretical Analysis of Soft-Label vs Hard-Label Training in Neural Networks—0
Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models—0
Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation—0
Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection—0
Unsupervised Continual Learning Via Pseudo Labels—0
Continual Learning with Diffusion-based Generative Replay for Industrial Streaming Data—0
A Note on Knowledge Distillation Loss Function for Object Classification—0
Continual Learning with Dirichlet Generative-based Rehearsal—0
Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT—0
Extending Label Smoothing Regularization with Self-Knowledge Distillation—0
Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation—0
Extracting knowledge from features with multilevel abstraction—0
Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment—0
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation—0
Enhancing CTC-Based Visual Speech Recognition—0
Show:102550
← PrevPage 30 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified