SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 801–850 of 4240 papers

TitleStatusHype
Cross-category Video Highlight Detection via Set-based LearningCode1
Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltyCode1
It's All In the Teacher: Zero-Shot Quantization Brought Closer to the TeacherCode1
JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis ModelsCode1
Data-Free Knowledge Distillation for Heterogeneous Federated LearningCode1
itKD: Interchange Transfer-based Knowledge Distillation for 3D Object DetectionCode1
Fcaformer: Forward Cross Attention in Hybrid Vision TransformerCode1
Cross-Layer Distillation with Semantic CalibrationCode1
Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot ClassificationCode1
FreeKD: Knowledge Distillation via Semantic Frequency PromptCode1
Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose EstimationCode1
Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?Code1
From My View to Yours: Ego-Augmented Learning in Large Vision Language Models for Understanding Exocentric Daily Living ActivitiesCode1
It's All in the Head: Representation Knowledge Distillation through Classifier SharingCode1
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video RetrievalCode1
CrossMatch: Enhance Semi-Supervised Medical Image Segmentation with Perturbation Strategies and Knowledge DistillationCode1
Balanced Knowledge Distillation for Long-tailed LearningCode1
Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image RetrievalCode1
Cross-modality Data Augmentation for End-to-End Sign Language TranslationCode1
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine TranslationCode1
Cross-Modality Knowledge Distillation Network for Monocular 3D Object DetectionCode1
A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect DetectionCode1
Generative Adversarial Super-Resolution at the Edge with Knowledge DistillationCode1
Generative Model-based Feature Knowledge Distillation for Action RecognitionCode1
RankFormer: Listwise Learning-to-Rank Using Listwide LabelsCode1
Generative Bias for Robust Visual Question AnsweringCode1
Aligned Structured Sparsity Learning for Efficient Image Super-ResolutionCode1
Generic-to-Specific Distillation of Masked AutoencodersCode1
GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small DatasetsCode1
CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image FusionCode1
Intra-class Feature Variation Distillation for Semantic SegmentationCode1
CTC-based Non-autoregressive Textless Speech-to-Speech TranslationCode1
GlobalFlowNet: Video Stabilization using Deep Distilled Global Motion EstimatesCode1
Reference Twice: A Simple and Unified Baseline for Few-Shot Instance SegmentationCode1
Cumulative Spatial Knowledge Distillation for Vision TransformersCode1
Curriculum Learning for Dense Retrieval DistillationCode1
Curriculum Temperature for Knowledge DistillationCode1
BearingPGA-Net: A Lightweight and Deployable Bearing Fault Diagnosis Network via Decoupled Knowledge Distillation and FPGA AccelerationCode1
Go From the General to the Particular: Multi-Domain Translation with Domain Transformation NetworksCode1
Gradient-based Intra-attention Pruning on Pre-trained Language ModelsCode1
Remembering Normality: Memory-guided Knowledge Distillation for Unsupervised Anomaly DetectionCode1
Representation Compensation Networks for Continual Semantic SegmentationCode1
Grad-CAM++: Improved Visual Explanations for Deep Convolutional NetworksCode1
Good Teachers Explain: Explanation-Enhanced Knowledge DistillationCode1
Rethinking Centered Kernel Alignment in Knowledge DistillationCode1
Rethinking Data Augmentation for Robust Visual Question AnsweringCode1
Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language ModelCode1
Graph-based Knowledge Distillation: A survey and experimental evaluationCode1
Dark Experience for General Continual Learning: a Strong, Simple BaselineCode1
Data-Free Class-Incremental Hand Gesture RecognitionCode1
Show:102550
← PrevPage 17 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified