SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3801–3850 of 4240 papers

TitleStatusHype
Knowledge Distillation for Anomaly Detection—0
Knowledge Distillation for Bilingual Dictionary Induction—0
Knowledge Distillation of Black-Box Large Language Models—0
Knowledge Distillation for Efficient Sequences of Training Runs—0
Knowledge Distillation for Efficient Audio-Visual Video Captioning—0
Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models—0
Knowledge distillation for fast and accurate DNA sequence correction—0
Knowledge Distillation for Federated Learning: a Practical Guide—0
Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images—0
Knowledge Distillation for Improved Accuracy in Spoken Question Answering—0
Knowledge Distillation for Incremental Learning in Semantic Segmentation—0
Knowledge Distillation for Mobile Edge Computation Offloading—0
Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation—0
Knowledge Distillation for Multimodal Egocentric Action Recognition Robust to Missing Modalities—0
Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data—0
Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models—0
Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation—0
Knowledge Distillation for Object Detection: from generic to remote sensing datasets—0
Knowledge Distillation for Oriented Object Detection on Aerial Images—0
Knowledge Distillation for Real-Time Classification of Early Media in Voice Communications—0
Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization—0
Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition—0
Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning—0
Knowledge distillation for semi-supervised domain adaptation—0
Knowledge Distillation for Small-footprint Highway Networks—0
Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance—0
Knowledge Distillation for Sustainable Neural Machine Translation—0
Knowledge Distillation for Swedish NER models: A Search for Performance and Efficiency—0
Knowledge Distillation for Underwater Feature Extraction and Matching via GAN-synthesized Images—0
Knowledge Distillation Framework for Accelerating High-Accuracy Neural Network-Based Molecular Dynamics Simulations—0
Boosting of Head Pose Estimation by Knowledge Distillation—0
Knowledge Distillation from Few Samples—0
Knowledge Distillation from Internal Representations—0
Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach—0
Knowledge Distillation from Language-Oriented to Emergent Communication for Multi-Agent Remote Control—0
Knowledge distillation from multi-modal to mono-modal segmentation networks—0
Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition—0
Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer—0
Knowledge Distillation Improves Stability in Retranslation-based Simultaneous Translation—0
Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels—0
Knowledge Distillation in Deep Learning and its Applications—0
Knowledge Distillation in Document Retrieval—0
Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions—0
Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students—0
Knowledge Distillation in Vision Transformers: A Critical Review—0
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher—0
Knowledge Distillation Meets Few-Shot Learning: An Approach for Few-Shot Intent Classification Within and Across Domains—0
Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains—0
Knowledge Distillation Neural Network for Predicting Car-following Behaviour of Human-driven and Autonomous Vehicles—0
Knowledge Distillation of Convolutional Neural Networks through Feature Map Transformation using Decision Trees—0
Show:102550
← PrevPage 77 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified