SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3701–3750 of 4240 papers

TitleStatusHype
Distilling the Undistillable: Learning from a Nasty TeacherCode0
Tiny Updater: Towards Efficient Neural Network-Driven Software UpdatingCode0
AdaGMLP: AdaBoosting GNN-to-MLP Knowledge DistillationCode0
Induced Model Matching: Restricted Models Help Train Full-Featured ModelsCode0
Induced Model Matching: How Restricted Models Can Help Larger OnesCode0
Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision TransformersCode0
Spatial-Channel Token Distillation for Vision MLPsCode0
Masked Student Dataset of ExpressionsCode0
InDistill: Information flow-preserving knowledge distillation for model compressionCode0
COMBHelper: A Neural Approach to Reduce Search Space for Graph Combinatorial ProblemsCode0
Distilling Knowledge by Mimicking FeaturesCode0
Reciprocal Supervised Learning Improves Neural Machine TranslationCode0
Why does Knowledge Distillation Work? Rethink its Attention and Fidelity MechanismCode0
Incremental Meta-Learning via Episodic Replay Distillation for Few-Shot Image RecognitionCode0
MCC-KD: Multi-CoT Consistent Knowledge DistillationCode0
To Distill or Not to Distill? On the Robustness of Robust Knowledge DistillationCode0
UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit RepresentationCode0
Incorporating Graph Information in Transformer-based AMR ParsingCode0
An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question AnsweringCode0
Improving Stance Detection with Multi-Dataset Learning and Knowledge DistillationCode0
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from EnsemblesCode0
Spatio-Temporal Branching for Motion Prediction using Motion IncrementsCode0
Improving Question Answering Performance Using Knowledge Distillation and Active LearningCode0
MedDet: Generative Adversarial Distillation for Efficient Cervical Disc Herniation DetectionCode0
AI-KD: Towards Alignment Invariant Face Image Quality Assessment Using Knowledge DistillationCode0
Distilling the Knowledge of Romanian BERTs Using Multiple TeachersCode0
Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain ConversationCode0
Unsupervised Domain Expansion for Visual CategorizationCode0
Improving Neural Topic Models with Wasserstein Knowledge DistillationCode0
SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral AnalysisCode0
Improving Neural Architecture Search Image Classifiers via Ensemble LearningCode0
MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context LearningCode0
Redefining Normal: A Novel Object-Level Approach for Multi-Object Novelty DetectionCode0
Improving Knowledge Distillation via Transferring Learning AbilityCode0
Adaptive Prompt Learning with Distilled Connective Knowledge for Implicit Discourse Relation RecognitionCode0
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive TranslationCode0
Reducing Capacity Gap in Knowledge Distillation with Review Mechanism for Crowd CountingCode0
Improving generalizability of distilled self-supervised speech processing models under distorted settingsCode0
Reducing Spatial Fitting Error in Distillation of Denoising Diffusion ModelsCode0
Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic TranscriptsCode0
Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge DistillationCode0
Autoregressive Knowledge Distillation through Imitation LearningCode0
An Efficient Memory Module for Graph Few-Shot Class-Incremental LearningCode0
Improving Robustness by Enhancing Weak SubnetsCode0
Improving Adversarial Robust Fairness via Anti-Bias Soft Label DistillationCode0
Improved Knowledge Distillation via Teacher AssistantCode0
Collective Relevance Labeling for Passage RetrievalCode0
Improved Knowledge Distillation for Crowd Counting on IoT DeviceCode0
IE-GAN: An Improved Evolutionary Generative Adversarial Network Using a New Fitness Function and a Generic Crossover OperatorCode0
Distilling Stereo Networks for Performant and Efficient Leaner NetworksCode0
Show:102550
← PrevPage 75 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified