SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2701–2725 of 4240 papers

TitleStatusHype
A "Network Pruning Network" Approach to Deep Model Compression—0
A New Method to Capturing Compositional Knowledge in Linguistic Space—0
An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits—0
An Interpretable Neuron Embedding for Static Knowledge Distillation—0
A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss—0
A Novel Approach To Implementing Knowledge Distillation In Tsetlin Machines—0
A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation—0
A novel channel pruning method for deep neural network compression—0
A Novel Garment Transfer Method Supervised by Distilled Knowledge of Virtual Try-on Model—0
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning—0
A Novel Local-Global Feature Fusion Framework for Body-weight Exercise Recognition with Pressure Mapping Sensors—0
A Novel Self-Knowledge Distillation Approach with Siamese Representation Learning for Action Recognition—0
A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation—0
An Overview of Neural Network Compression—0
AntMan: Sparse Low-Rank Compression to Accelerate RNN inference—0
An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition—0
APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks—0
A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts—0
A Plasticity-Aware Method for Continual Self-Supervised Learning in Remote Sensing—0
Application of Knowledge Distillation to Multi-task Speech Representation Learning—0
Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving—0
Applications of Knowledge Distillation in Remote Sensing: A Survey—0
Applied Federated Model Personalisation in the Industrial Domain: A Comparative Study—0
Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition—0
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy—0
Show:102550
← PrevPage 109 of 170Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified