SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1501–1525 of 4240 papers

TitleStatusHype
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image ClassificationCode0
Guiding Frame-Level CTC Alignments Using Self-knowledge DistillationCode0
GSB: Group Superposition Binarization for Vision Transformer with Limited Training SamplesCode0
Group Multi-View Transformer for 3D Shape Analysis with Spatial EncodingCode0
GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric LearningCode0
On the Generalization vs Fidelity Paradox in Knowledge DistillationCode0
Facial Landmark Points Detection Using Knowledge Distillation-Based Neural NetworksCode0
Facilitating NSFW Text Detection in Open-Domain Dialogue Systems via Knowledge DistillationCode0
Facilitating Pornographic Text Detection for Open-Domain Dialogue Systems via Knowledge Distillation of Large Language ModelsCode0
On the Transferability of Visual Features in Generalized Zero-Shot LearningCode0
Greedy-layer Pruning: Speeding up Transformer Models for Natural Language ProcessingCode0
Distill n' Explain: explaining graph neural networks using simple surrogatesCode0
Graph Knowledge Distillation to Mixture of ExpertsCode0
Distilling Virtual Examples for Long-tailed RecognitionCode0
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series DataCode0
A Dual-Contrastive Framework for Low-Resource Cross-Lingual Named Entity RecognitionCode0
Distilling the Undistillable: Learning from a Nasty TeacherCode0
Graph Entropy Minimization for Semi-supervised Node ClassificationCode0
Distilling the Knowledge of Romanian BERTs Using Multiple TeachersCode0
Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain ConversationCode0
CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature SharingCode0
An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity RecognitionCode0
Gradient Knowledge Distillation for Pre-trained Language ModelsCode0
FAKD: Feature Augmented Knowledge Distillation for Semantic SegmentationCode0
GOTHAM: Graph Class Incremental Learning Framework under Weak SupervisionCode0
Show:102550
← PrevPage 61 of 170Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified