SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2901–2950 of 4240 papers

TitleStatusHype
Classroom-Inspired Multi-Mentor Distillation with Adaptive Learning Strategies—0
Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation—0
Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems—0
Collaborative Learning for Enhanced Unsupervised Domain Adaptation—0
CleverDistiller: Simple and Spatially Consistent Cross-modal Distillation—0
CLFace: A Scalable and Resource-Efficient Continual Learning Framework for Lifelong Face Recognition—0
Client Clustering Meets Knowledge Sharing: Enhancing Privacy and Robustness in Personalized Peer-to-Peer Learning—0
CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination—0
CLIPPING: Distilling CLIP-Based Models With a Student Base for Video-Language Retrieval—0
CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs—0
Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning—0
Cloud-Device Collaborative Learning for Multimodal Large Language Models—0
CL-ReKD: Cross-lingual Knowledge Distillation for Multilingual Retrieval Question Answering—0
ClST: A Convolutional Transformer Framework for Automatic Modulation Recognition by Knowledge Distillation—0
CMU’s IWSLT 2022 Dialect Speech Translation System—0
CoCo DistillNet: a Cross-layer Correlation Distillation Network for Pathological Gastric Cancer Segmentation—0
CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition—0
Cold & Warm Net: Addressing Cold-Start Users in Recommender Systems—0
Collaborative Distillation for Top-N Recommendation—0
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition—0
Collaborative Inter-agent Knowledge Distillation for Reinforcement Learning—0
Collaborative Learning for Deep Neural Networks—0
Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks—0
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer—0
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders—0
Collective Knowledge Graph Completion with Mutual Knowledge Distillation—0
Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation—0
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks—0
Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation—0
CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation—0
ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model—0
Batch Selection and Communication for Active Learning with Edge Labeling—0
Compact CNN Models for On-device Ocular-based User Recognition in Mobile Devices—0
Compact CNN Structure Learning by Knowledge Distillation—0
Compacting Deep Neural Networks for Internet of Things: Methods and Applications—0
Compact Speaker Embedding: lrx-vector—0
Comparing Fisher Information Regularization with Distillation for DNN Quantization—0
Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR—0
Completely Heterogeneous Federated Learning—0
Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning—0
Complex Emotion Recognition System using basic emotions via Facial Expression, EEG, and ECG Signals: a review—0
Compositional Data Augmentation for Abstractive Conversation Summarization—0
Comprehensive Pathological Image Segmentation via Teacher Aggregation for Tumor Microenvironment Analysis—0
Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models—0
Comprehensive Survey of Model Compression and Speed up for Vision Transformers—0
Compressed Meta-Optical Encoder for Image Classification—0
Compressing Deep Image Super-resolution Models—0
Compressing GANs using Knowledge Distillation—0
Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold—0
Compressing Recurrent Neural Networks for FPGA-accelerated Implementation in Fluorescence Lifetime Imaging—0
Show:102550
← PrevPage 59 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified