SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 901–950 of 4240 papers

TitleStatusHype
BiLD: Bi-directional Logits Difference Loss for Large Language Model DistillationCode1
Simplified TinyBERT: Knowledge Distillation for Document RetrievalCode1
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video RetrievalCode1
SLMRec: Distilling Large Language Models into Small for Sequential RecommendationCode1
A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect DetectionCode1
Decoupled Multimodal Distilling for Emotion RecognitionCode1
Does Knowledge Distillation Really Work?Code1
A Neural Span-Based Continual Named Entity Recognition ModelCode1
Domain Generalization for Prostate Segmentation in Transrectal Ultrasound Images: A Multi-center StudyCode1
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech ModelsCode1
DM-VTON: Distilled Mobile Real-time Virtual Try-OnCode1
DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge DistillationCode1
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit DifferentiationCode1
DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any ArchitectureCode1
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and RetrievalCode1
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic HashingCode1
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine TranslationCode1
AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian DetectionCode1
Structured Sparse R-CNN for Direct Scene Graph GenerationCode1
Deep Graph-level Anomaly Detection by Glocal Knowledge DistillationCode1
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation TrainerCode1
Black-Box Attacks on Sequential Recommenders via Data-Free Model ExtractionCode1
SUOD: Toward Scalable Unsupervised Outlier DetectionCode1
Adaptive Multi-Teacher Knowledge Distillation with Meta-LearningCode1
Supervised Compression for Resource-Constrained Edge Computing SystemsCode1
Black-box Few-shot Knowledge DistillationCode1
Distill on the Go: Online knowledge distillation in self-supervised learningCode1
DIOD: Self-Distillation Meets Object DiscoveryCode1
Tailoring Instructions to Student's Learning Levels Boosts Knowledge DistillationCode1
Distilling a Powerful Student Model via Online Knowledge DistillationCode1
Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance SegmentationCode1
Teachers Do More Than Teach: Compressing Image-to-Image ModelsCode1
Deep Structured Instance Graph for Distilling Object DetectorsCode1
Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge DistillationCode1
Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine TranslationCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Temporal Self-Ensembling Teacher for Semi-Supervised Object DetectionCode1
Defocus Blur Detection via Depth DistillationCode1
Deformation Flow Based Two-Stream Network for Lip ReadingCode1
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge DistillationCode1
Deliberated Domain Bridging for Domain Adaptive Semantic SegmentationCode1
Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge GraphsCode1
DistilPose: Tokenized Pose Regression with Heatmap DistillationCode1
DPM-OT: A New Diffusion Probabilistic Model Based on Optimal TransportCode1
EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic DataCode1
End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge DistillationCode1
FerKD: Surgical Label Adaptation for Efficient DistillationCode1
Dense Interspecies Face EmbeddingCode1
Instance-Conditional Knowledge Distillation for Object DetectionCode1
Show:102550
← PrevPage 19 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified