SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3651–3700 of 4240 papers

TitleStatusHype
Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge GraphsCode0
Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal SummarizationCode0
DMSSN: Distilled Mixed Spectral-Spatial Network for Hyperspectral Salient Object DetectionCode0
TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth EstimationCode0
A Knowledge Distillation Ensemble Framework for Predicting Short and Long-term Hospitalisation Outcomes from Electronic Health Records DataCode0
SMOTExT: SMOTE meets Large Language ModelsCode0
Random Path Selection for Continual LearningCode0
An Adaptive Random Path Selection Approach for Incremental LearningCode0
A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier ModelsCode0
Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation ModelCode0
Joint Answering and Explanation for Visual Commonsense ReasoningCode0
Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning FrameworkCode0
Comparative Knowledge DistillationCode0
Is Modularity Transferable? A Case Study through the Lens of Knowledge DistillationCode0
Distribution Aligned Semantics Adaption for Lifelong Person Re-IdentificationCode0
Low-Complexity Acoustic Scene Classification Using Parallel Attention-Convolution NetworkCode0
BEBERT: Efficient and Robust Binary Ensemble BERTCode0
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender SystemCode0
Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge DistillationCode0
Invariant debiasing learning for recommendation via biased imputationCode0
Adaptive Teaching with Shared Classifier for Knowledge DistillationCode0
Distributed Soft Actor-Critic with Multivariate Reward Representation and Knowledge DistillationCode0
Compact Trilinear Interaction for Visual Question AnsweringCode0
CoMoTo: Unpaired Cross-Modal Lesion Distillation Improves Breast Lesion Detection in TomosynthesisCode0
Adaptive Search-and-Training for Robust and Efficient Network PruningCode0
Tiny models from tiny data: Textual and null-text inversion for few-shot distillationCode0
Intra-class Patch Swap for Self-DistillationCode0
Distill n' Explain: explaining graph neural networks using simple surrogatesCode0
Interpreting Microbiome Relative Abundance Data Using Symbolic RegressionCode0
Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidanceCode0
RDPD: Rich Data Helps Poor Data via ImitationCode0
Interpreting and Disentangling Feature Components of Various Complexity from DNNsCode0
Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural NetworkCode0
BAM! Born-Again Multi-Task Networks for Natural Language UnderstandingCode0
m2mKD: Module-to-Module Knowledge Distillation for Modular TransformersCode0
3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document UnderstandingCode0
Distilling Virtual Examples for Long-tailed RecognitionCode0
Inter-Domain Alignment for Predicting High-Resolution Brain Networks Using Teacher-Student LearningCode0
Instance Temperature Knowledge DistillationCode0
Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic SegmentationCode0
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review MechanismCode0
Real-Time Cell Sorting with Scalable In Situ FPGA-Accelerated Deep LearningCode0
Real-Time Correlation Tracking via Joint Model Compression and TransferCode0
Real-Time Decentralized knowledge Transfer at the EdgeCode0
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series DataCode0
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric AnnotationsCode0
Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial BiasCode0
AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual LearningCode0
Aligning Logits Generatively for Principled Black-Box Knowledge DistillationCode0
Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge DistillationCode0
Show:102550
← PrevPage 74 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified