SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 901950 of 4240 papers

TitleStatusHype
Language Model Prior for Low-Resource Neural Machine TranslationCode1
Distilling Knowledge from Refinement in Multiple Instance Detection NetworksCode1
Making Monolingual Sentence Embeddings Multilingual using Knowledge DistillationCode1
Role-Wise Data Augmentation for Knowledge DistillationCode1
Triplet Loss for Knowledge DistillationCode1
Multimodal and multiview distillation for real-time player detection on a football fieldCode1
Dark Experience for General Continual Learning: a Strong, Simple BaselineCode1
Inter-Region Affinity Distillation for Road Marking SegmentationCode1
KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflowCode1
Structure-Level Knowledge Distillation For Multilingual Sequence LabelingCode1
On the Effect of Dropping Layers of Pre-trained Transformer ModelsCode1
Towards Efficient Unconstrained Palmprint Recognition via Deep Distillation HashingCode1
Temporally Distributed Networks for Fast Video Semantic SegmentationCode1
More Grounded Image Captioning by Distilling Image-Text Matching ModelCode1
Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal HashingCode1
Regularizing Class-wise Predictions via Self-knowledge DistillationCode1
Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox ModelCode1
Distilled Semantics for Comprehensive Scene Understanding from VideosCode1
Circumventing Outliers of AutoAugment with Knowledge DistillationCode1
Distilling Knowledge from Graph Convolutional NetworksCode1
Collaborative Distillation for Ultra-Resolution Universal Style TransferCode1
Incremental Object Detection via Meta-LearningCode1
Deformation Flow Based Two-Stream Network for Lip ReadingCode1
SuperMix: Supervising the Mixing Data AugmentationCode1
Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNNCode1
PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge DistillationCode1
Efficient Semantic Video Segmentation with Per-frame InferenceCode1
Knapsack Pruning with Inner DistillationCode1
Salvaging Federated Learning by Local AdaptationCode1
Knowledge Distillation for Brain Tumor SegmentationCode1
SUOD: Toward Scalable Unsupervised Outlier DetectionCode1
BERT-of-Theseus: Compressing BERT by Progressive Module ReplacingCode1
Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed ClassificationCode1
Unpaired Multi-modal Segmentation via Knowledge DistillationCode1
Blockwisely Supervised Neural Architecture Search with Knowledge DistillationCode1
Go From the General to the Particular: Multi-Domain Translation with Domain Transformation NetworksCode1
Preparing Lessons: Improve Knowledge Distillation with Better SupervisionCode1
Maintaining Discrimination and Fairness in Class Incremental LearningCode1
Learning from a Teacher using Unlabeled DataCode1
Data Diversification: A Simple Strategy For Neural Machine TranslationCode1
Contrastive Representation DistillationCode1
FedMD: Heterogenous Federated Learning via Model DistillationCode1
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterCode1
Distilled Split Deep Neural Networks for Edge-Assisted Real-Time SystemsCode1
Distillation-Based Training for Multi-Exit ArchitecturesCode1
Improved Techniques for Training Adaptive Deep NetworksCode1
When Does Label Smoothing Help?Code1
Adversarially Robust DistillationCode1
Knowledge Distillation via Route Constrained OptimizationCode1
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary CellsCode1
Show:102550
← PrevPage 19 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified