SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2701–2750 of 4240 papers

TitleStatusHype
A "Network Pruning Network" Approach to Deep Model Compression—0
A New Method to Capturing Compositional Knowledge in Linguistic Space—0
An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits—0
An Interpretable Neuron Embedding for Static Knowledge Distillation—0
A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss—0
A Novel Approach To Implementing Knowledge Distillation In Tsetlin Machines—0
A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation—0
A novel channel pruning method for deep neural network compression—0
A Novel Garment Transfer Method Supervised by Distilled Knowledge of Virtual Try-on Model—0
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning—0
A Novel Local-Global Feature Fusion Framework for Body-weight Exercise Recognition with Pressure Mapping Sensors—0
A Novel Self-Knowledge Distillation Approach with Siamese Representation Learning for Action Recognition—0
A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation—0
An Overview of Neural Network Compression—0
AntMan: Sparse Low-Rank Compression to Accelerate RNN inference—0
An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition—0
APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks—0
A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts—0
A Plasticity-Aware Method for Continual Self-Supervised Learning in Remote Sensing—0
Application of Knowledge Distillation to Multi-task Speech Representation Learning—0
Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving—0
Applications of Knowledge Distillation in Remote Sensing: A Survey—0
Applied Federated Model Personalisation in the Industrial Domain: A Comparative Study—0
Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition—0
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy—0
Apprentissage automatique de repr\'esentation de voix \`a l'aide d'une distillation de la connaissance pour le casting vocal (Learning voice representation using knowledge distillation for automatic voice casting )—0
A Practical Survey on Faster and Lighter Transformers—0
A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene—0
ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation—0
A Recipe for Efficient SBIR Models: Combining Relative Triplet Loss with Batch Normalization and Knowledge Distillation—0
A Review on Discriminative Self-supervised Learning Methods in Computer Vision—0
Artificial Behavior Intelligence: Technology, Challenges, and Future Directions—0
A scalable convolutional neural network for task-specified scenarios via knowledge distillation—0
A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models—0
A Short Study on Compressing Decoder-Based Language Models—0
A Simple and Generic Framework for Feature Distillation via Channel-wise Transformation—0
A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems—0
SS-IL: Separated Softmax for Incremental Learning—0
A Simple Linear Patch Revives Layer-Pruned Large Language Models—0
A Simple Recipe for Competitive Low-compute Self supervised Vision Models—0
Asterisk*: Keep it Simple—0
A Study of Non-autoregressive Model for Sequence Generation—0
A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models—0
A Study on the Efficiency and Generalization of Light Hybrid Retrievers—0
A Survey of Methods for Low-Power Deep Learning and Computer Vision—0
A Survey of Model Compression and Acceleration for Deep Neural Networks—0
A Survey of Techniques for Optimizing Transformer Inference—0
A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions—0
A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking—0
A Survey on Green Deep Learning—0
Show:102550
← PrevPage 55 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified