SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2951–3000 of 4240 papers

TitleStatusHype
Continual Learning for Class- and Domain-Incremental Semantic Segmentation—0
CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation—0
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation—0
PlaStIL: Plastic and Stable Memory-Free Class-Incremental Learning—0
TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation—0
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity MatchingCode0
Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation—0
Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation—0
Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning—0
In-situ animal behavior classification using knowledge distillation and fixed-point quantization—0
Exploring Target Representations for Masked AutoencodersCode0
ViTKD: Practical Guidelines for ViT feature knowledge distillation—0
A Novel Self-Knowledge Distillation Approach with Siamese Representation Learning for Action Recognition—0
Knowledge Distillation for Sustainable Neural Machine Translation—0
Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization—0
FAKD: Feature Augmented Knowledge Distillation for Semantic SegmentationCode0
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning StrategyCode0
Removing Rain Streaks via Task Transfer Learning—0
Goal-Conditioned Q-Learning as Knowledge DistillationCode0
Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge TransferCode0
Dense Depth Distillation with Out-of-Distribution Simulated Images—0
Debias the Black-box: A Fair Ranking Framework via Knowledge Distillation—0
Lifelong Learning for Neural powered Mixed Integer Programming—0
FS-BAN: Born-Again Networks for Domain Generalization Few-Shot ClassificationCode0
Multi-View Attention Transfer for Efficient Speech Enhancement—0
Rethinking Knowledge Distillation via Cross-Entropy—0
RAIN: RegulArization on Input and Network for Black-Box Domain AdaptationCode0
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks—0
Effectiveness of Function Matching in Driving Scene Recognition—0
Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification—0
Leukocyte Classification using Multimodal Architecture Enhanced by Knowledge Distillation—0
Progressive Cross-modal Knowledge Distillation for Human Action Recognition—0
Unsupervised Domain Adaptation for Segmentation with Black-box Source Model—0
RAWtoBit: A Fully End-to-end Camera ISP Network—0
Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model FusionCode0
A Knowledge Distillation-Based Backdoor Attack in Federated Learning—0
Non-Autoregressive Sign Language Production via Knowledge Distillation—0
BEiT v2: Masked Image Modeling with Vector-Quantized Visual TokenizersCode0
Self-Knowledge Distillation via Dropout—0
SKDCGN: Source-free Knowledge Distillation of Counterfactual Generative Networks using cGANsCode0
Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation—0
Study of Encoder-Decoder Architectures for Code-Mix Search Query Translation—0
PGX: A Multi-level GNN Explanation Framework Based on Separate Knowledge Distillation Processes—0
Task-Balanced Distillation for Object Detection—0
Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection—0
Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer—0
SDBERT: SparseDistilBERT, a faster and smaller BERT model—0
NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation—0
Exploring Generalizable Distillation for Efficient Medical Image SegmentationCode0
Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations—0
Show:102550
← PrevPage 60 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified