SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 751800 of 4240 papers

TitleStatusHype
Knowledge Inheritance for Pre-trained Language ModelsCode1
Selective Knowledge Distillation for Neural Machine TranslationCode1
Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' OutputsCode1
Backdoor Attacks on Self-Supervised LearningCode1
Intra-Document Cascading: Learning to Select Passages for Neural Document RankingCode1
Data-Free Knowledge Distillation for Heterogeneous Federated LearningCode1
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge DistillationCode1
Contrastive Model Inversion for Data-Free Knowledge DistillationCode1
Graph-Free Knowledge Distillation for Graph Neural NetworksCode1
Undistillable: Making A Nasty Teacher That CANNOT teach studentsCode1
AgeFlow: Conditional Age Progression and Regression with Normalizing FlowsCode1
Boosting Light-Weight Depth Estimation Via Knowledge DistillationCode1
When Human Pose Estimation Meets Robustness: Adversarial Algorithms and BenchmarksCode1
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge DistillationCode1
Initialization and Regularization of Factorized Neural LayersCode1
Open-vocabulary Object Detection via Vision and Language Knowledge DistillationCode1
Distilling Audio-Visual Knowledge by Compositional Contrastive LearningCode1
Balanced Knowledge Distillation for Long-tailed LearningCode1
Voice2Mesh: Cross-Modal 3D Face Model Generation from VoicesCode1
Distill on the Go: Online knowledge distillation in self-supervised learningCode1
Distilling Knowledge via Knowledge ReviewCode1
On Learning the Geodesic Path for Incremental LearningCode1
Ego-Exo: Transferring Visual Representations from Third-person to First-person VideosCode1
Counter-Interference Adapter for Multilingual Machine TranslationCode1
Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain TransferCode1
Class-Balanced Distillation for Long-Tailed Visual RecognitionCode1
Content-Aware GAN CompressionCode1
HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast ImagesCode1
Complementary Relation Contrastive DistillationCode1
Embedding Transfer with Label Relaxation for Improved Metric LearningCode1
Multimodal Knowledge ExpansionCode1
Distilling Object Detectors via Decoupled FeaturesCode1
Distilling a Powerful Student Model via Online Knowledge DistillationCode1
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine TranslationCode1
ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquesCode1
Self-Supervised Adaptation for Video Super-ResolutionCode1
Human-Inspired Multi-Agent Navigation using Knowledge DistillationCode1
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge DistillationCode1
Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve BackbonesCode1
Parser-Free Virtual Try-on via Distilling Appearance FlowsCode1
Adaptive Multi-Teacher Multi-level Knowledge DistillationCode1
Distributed Dynamic Map Fusion via Federated Learning for Intelligent Networked VehiclesCode1
Teachers Do More Than Teach: Compressing Image-to-Image ModelsCode1
Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation FrameworkCode1
General Instance Distillation for Object DetectionCode1
Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot LearningCode1
Distilling Knowledge via Intermediate ClassifiersCode1
Training Generative Adversarial Networks in One StageCode1
Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-DistillationCode1
Localization Distillation for Dense Object DetectionCode1
Show:102550
← PrevPage 16 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified