SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 401450 of 4240 papers

TitleStatusHype
Decoupled Kullback-Leibler Divergence LossCode1
Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?Code1
DisCo: Distilled Student Models Co-training for Semi-supervised Text MiningCode1
Lifting the Curse of Capacity Gap in Distilling Language ModelsCode1
Cross-modality Data Augmentation for End-to-End Sign Language TranslationCode1
AD-KD: Attribution-Driven Knowledge Distillation for Language Model CompressionCode1
Tailoring Instructions to Student's Learning Levels Boosts Knowledge DistillationCode1
Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation ModelsCode1
Serial Contrastive Knowledge Distillation for Continual Few-shot Relation ExtractionCode1
Improving Continual Relation Extraction by Distinguishing Analogous SemanticsCode1
FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise HeterogeneityCode1
SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language ModelsCode1
Distilling Script Knowledge from Large Language Models for Constrained Language PlanningCode1
Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with UncertaintyCode1
SCOTT: Self-Consistent Chain-of-Thought DistillationCode1
DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge DistillationCode1
A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question AnsweringCode1
Class Attention Transfer Based Knowledge DistillationCode1
Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic SegmentationCode1
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual GraphsCode1
Attention Weighted Local DescriptorsCode1
OVTrack: Open-Vocabulary Multiple Object TrackingCode1
Robust Cross-Modal Knowledge Distillation for Unconstrained VideosCode1
Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation LearningCode1
Continual Learning for LiDAR Semantic Segmentation: Class-Incremental and Coarse-to-Fine strategies on Sparse DataCode1
DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic SegmentationCode1
Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good TeacherCode1
Knowledge Distillation for Feature Extraction in Underwater VSLAMCode1
Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuningCode1
SimDistill: Simulated Multi-modal Distillation for BEV 3D Object DetectionCode1
DisWOT: Student Architecture Search for Distillation WithOut TrainingCode1
HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language ModelsCode1
Dice Semimetric Losses: Optimizing the Dice Score with Soft LabelsCode1
UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye ViewCode1
Preserving Linear Separability in Continual Learning by Backward Feature ProjectionCode1
Supervised Masked Knowledge Distillation for Few-Shot TransformersCode1
CCL: Continual Contrastive Learning for LiDAR Place RecognitionCode1
Decoupled Multimodal Distilling for Emotion RecognitionCode1
Understanding the Role of the Projector in Knowledge DistillationCode1
AdaptGuard: Defending Against Universal Attacks for Model AdaptationCode1
Channel-Aware Distillation Transformer for Depth Estimation on Nano DronesCode1
Prototype Knowledge Distillation for Medical Segmentation with Missing ModalityCode1
TeSLA: Test-Time Self-Learning With Automatic Adversarial AugmentationCode1
Global Knowledge Calibration for Fast Open-Vocabulary SegmentationCode1
Action knowledge for video captioning with graph neural networksCode1
DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervisionCode1
Graph-less Collaborative FilteringCode1
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset ReinforcementCode1
SCPNet: Semantic Scene Completion on Point CloudCode1
Extending global-local view alignment for self-supervised learning with remote sensing imageryCode1
Show:102550
← PrevPage 9 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified