SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1301–1350 of 4240 papers

TitleStatusHype
Collaborative Deep Reinforcement LearningCode0
KDMOS:Knowledge Distillation for Motion SegmentationCode0
Joint Progressive Knowledge Distillation and Unsupervised Domain AdaptationCode0
Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge GraphsCode0
Few Sample Knowledge Distillation for Efficient Network CompressionCode0
Improved Knowledge Distillation via Full Kernel Matrix TransferCode0
Leveraging Large Language Models for Active Merchant Non-player CharactersCode0
Cogni-Net: Cognitive Feature Learning through Deep Visual PerceptionCode0
Invariant debiasing learning for recommendation via biased imputationCode0
Knowledge Distillation For Wireless Edge LearningCode0
Is Modularity Transferable? A Case Study through the Lens of Knowledge DistillationCode0
Adversarial Teacher-Student Representation Learning for Domain GeneralizationCode0
Intra-class Patch Swap for Self-DistillationCode0
Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural NetworkCode0
Interpreting and Disentangling Feature Components of Various Complexity from DNNsCode0
Efficient Multitask Dense Predictor via BinarizationCode0
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech RecognitionCode0
Inter-Domain Alignment for Predicting High-Resolution Brain Networks Using Teacher-Student LearningCode0
Interpreting Microbiome Relative Abundance Data Using Symbolic RegressionCode0
Instance Temperature Knowledge DistillationCode0
EaSyGuide : ESG Issue Identification Framework leveraging Abilities of Generative Large Language ModelsCode0
CL-XABSA: Contrastive Learning for Cross-lingual Aspect-based Sentiment AnalysisCode0
Assessor-Guided Learning for Continual EnvironmentsCode0
A Flexible Multi-Task Model for BERT ServingCode0
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review MechanismCode0
DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical ImagesCode0
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns ClusteringCode0
Induced Model Matching: Restricted Models Help Train Full-Featured ModelsCode0
InDistill: Information flow-preserving knowledge distillation for model compressionCode0
Efficient Ternary Weight Embedding Model: Bridging Scalability and PerformanceCode0
Distilling Knowledge by Mimicking FeaturesCode0
Induced Model Matching: How Restricted Models Can Help Larger OnesCode0
Dynamic Sub-graph Distillation for Robust Semi-supervised Continual LearningCode0
Knowledge Extraction with No Observable DataCode0
Incremental Meta-Learning via Episodic Replay Distillation for Few-Shot Image RecognitionCode0
PruMUX: Augmenting Data Multiplexing with Model CompressionCode0
Dynamic Rectification Knowledge DistillationCode0
Incorporating Graph Information in Transformer-based AMR ParsingCode0
UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit RepresentationCode0
Improving Question Answering Performance Using Knowledge Distillation and Active LearningCode0
Improving Neural Topic Models with Wasserstein Knowledge DistillationCode0
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from EnsemblesCode0
Closest Neighbors are Harmful for Lightweight Masked Auto-encodersCode0
Improving Neural Architecture Search Image Classifiers via Ensemble LearningCode0
3M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health DetectionCode0
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action RecognitionCode0
Improving Stance Detection with Multi-Dataset Learning and Knowledge DistillationCode0
Improving generalizability of distilled self-supervised speech processing models under distorted settingsCode0
Improving Robustness by Enhancing Weak SubnetsCode0
Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic TranscriptsCode0
Show:102550
← PrevPage 27 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified