SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 4101–4150 of 4240 papers

TitleStatusHype
An Adaptive Random Path Selection Approach for Incremental LearningCode0
Knowledge Distillation via Instance Relationship GraphCode0
On Knowledge distillation from complex networks for response prediction—0
Structured Knowledge Distillation for Semantic SegmentationCode0
Online Distilling from Checkpoints for Neural Machine Translation—0
SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient ModelsCode0
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation—0
Network Pruning via Transformable Architecture SearchCode0
Zero-Shot Knowledge Distillation in Deep NetworksCode0
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self DistillationCode0
Creating Lightweight Object Detectors with Model Compression for Deployment on Edge Devices—0
FEED: Feature-level Ensemble Effect for knowledge Distillation—0
Towards a better understanding of Vector Quantized Autoencoders—0
Semi-supervised Acoustic Event Detection based on tri-training—0
Segmenting the FutureCode0
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial NetworksCode0
Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System—0
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding—0
Feature Fusion for Online Mutual Knowledge DistillationCode0
Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation—0
End-to-End Speech Translation with Knowledge Distillation—0
Visual Relationship Detection with Language prior and SoftmaxCode0
Automatic adaptation of object detectors to new domains using self-trainingCode0
Examining the Mapping Functions of Denoising Autoencoders in Singing Voice Separation—0
Unifying Heterogeneous Classifiers with DistillationCode0
Improved training of binary networks for human pose estimation and image recognition—0
Variational Information Distillation for Knowledge Transfer—0
Knowledge Squeezed Adversarial Network Compression—0
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency—0
Relational Knowledge DistillationCode0
Ultrafast Video Attention Prediction with Coupled Knowledge Distillation—0
Knowledge Distillation for Human Action Anticipation—0
Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization—0
Long-Term Vehicle Localization by Recursive Knowledge Distillation—0
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion—0
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning—0
Correlation Congruence for Knowledge DistillationCode0
A Comprehensive Overhaul of Feature DistillationCode0
Making Neural Machine Reading Comprehension Faster—0
Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation—0
Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness—0
Improving Neural Architecture Search Image Classifiers via Ensemble LearningCode0
Knowledge Adaptation for Efficient Semantic Segmentation—0
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation—0
Structured Knowledge Distillation for Dense PredictionCode0
SeizureNet: Multi-Spectral Deep Feature Learning for Seizure Type ClassificationCode0
TKD: Temporal Knowledge Distillation for Active Perception—0
Multilingual Neural Machine Translation with Knowledge DistillationCode0
Improved Knowledge Distillation via Teacher AssistantCode0
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression—0
Show:102550
← PrevPage 83 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified