SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2551–2600 of 4240 papers

TitleStatusHype
RAIN: RegulArization on Input and Network for Black-Box Domain AdaptationCode0
Multi-Granularity Distillation Scheme Towards Lightweight Semi-Supervised Semantic SegmentationCode1
LTE4G: Long-Tail Experts for Graph Neural NetworksCode1
Rethinking Knowledge Distillation via Cross-Entropy—0
Multi-View Attention Transfer for Efficient Speech Enhancement—0
A semi-supervised Teacher-Student framework for surgical tool detection and localizationCode1
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks—0
Effectiveness of Function Matching in Driving Scene Recognition—0
Mind the Gap in Distilling StyleGANsCode1
Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification—0
Leukocyte Classification using Multimodal Architecture Enhanced by Knowledge Distillation—0
Progressive Cross-modal Knowledge Distillation for Human Action Recognition—0
Unsupervised Domain Adaptation for Segmentation with Black-box Source Model—0
Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model FusionCode0
RAWtoBit: A Fully End-to-end Camera ISP Network—0
BEiT v2: Masked Image Modeling with Vector-Quantized Visual TokenizersCode0
Non-Autoregressive Sign Language Production via Knowledge Distillation—0
A Knowledge Distillation-Based Backdoor Attack in Federated Learning—0
MixSKD: Self-Knowledge Distillation from Mixup for Image RecognitionCode1
Self-Knowledge Distillation via Dropout—0
PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge DistillationCode1
SKDCGN: Source-free Knowledge Distillation of Counterfactual Generative Networks using cGANsCode0
Study of Encoder-Decoder Architectures for Code-Mix Search Query Translation—0
Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation—0
Task-Balanced Distillation for Object Detection—0
PGX: A Multi-level GNN Explanation Framework Based on Separate Knowledge Distillation Processes—0
Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume PredictionCode1
Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection—0
KD-SCFNet: Towards More Accurate and Efficient Salient Object Detection via Knowledge DistillationCode1
Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer—0
Generative Bias for Robust Visual Question AnsweringCode1
Chinese grammatical error correction based on knowledge distillationCode1
Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage RetrievalCode1
SDBERT: SparseDistilBERT, a faster and smaller BERT model—0
Meta-Learning based Degradation Representation for Blind Super-ResolutionCode1
NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation—0
Exploring Generalizable Distillation for Efficient Medical Image SegmentationCode0
HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks—0
Domain-invariant Feature Exploration for Domain Generalization—0
Black-box Few-shot Knowledge DistillationCode1
Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations—0
Spatial-Channel Token Distillation for Vision MLPsCode0
Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and FusionCode0
Online Knowledge Distillation via Mutual Contrastive Learning for Visual RecognitionCode1
Hyper-Representations for Pre-Training and Transfer LearningCode1
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free ReplayCode0
Federated Semi-Supervised Domain Adaptation via Knowledge Transfer—0
TinyViT: Fast Pretraining Distillation for Small Vision Transformers—0
KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view StereoCode1
Many-to-One Knowledge Distillation of Real-Time Epileptic Seizure Detection for Low-Power Wearable Internet of Things Systems—0
Show:102550
← PrevPage 52 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified