SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 12511300 of 4240 papers

TitleStatusHype
Comparative Knowledge DistillationCode0
Compact Trilinear Interaction for Visual Question AnsweringCode0
Knowledge Distillation By Sparse Representation MatchingCode0
Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical DiagnosisCode0
Knowledge Distillation by On-the-Fly Native EnsembleCode0
Knowledge Distillation as Semiparametric InferenceCode0
Knowledge Distillation-Based Model Extraction Attack using GAN-based Private Counterfactual ExplanationsCode0
Knowledge Distillation approach towards Melanoma DetectionCode0
CoMoTo: Unpaired Cross-Modal Lesion Distillation Improves Breast Lesion Detection in TomosynthesisCode0
A Flexible Multi-Task Model for BERT ServingCode0
Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidanceCode0
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge DistillationCode0
Knowledge Distillation for Detection Transformer with Consistent Distillation Points SamplingCode0
COMBHelper: A Neural Approach to Reduce Search Space for Graph Combinatorial ProblemsCode0
KDMOS:Knowledge Distillation for Motion SegmentationCode0
Collective Relevance Labeling for Passage RetrievalCode0
Joint Progressive Knowledge Distillation and Unsupervised Domain AdaptationCode0
Joint Answering and Explanation for Visual Commonsense ReasoningCode0
Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep DeploymentCode0
Collaborative Learning of Bidirectional Decoders for Unsupervised Text Style TransferCode0
Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge GraphsCode0
Efficient and Robust Jet Tagging at the LHC with Knowledge DistillationCode0
Is Modularity Transferable? A Case Study through the Lens of Knowledge DistillationCode0
Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion SamplingCode0
Invariant debiasing learning for recommendation via biased imputationCode0
Interpreting Microbiome Relative Abundance Data Using Symbolic RegressionCode0
Collaborative Deep Reinforcement LearningCode0
Interpreting and Disentangling Feature Components of Various Complexity from DNNsCode0
Inter-Domain Alignment for Predicting High-Resolution Brain Networks Using Teacher-Student LearningCode0
Instance Temperature Knowledge DistillationCode0
Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural NetworkCode0
Intra-class Patch Swap for Self-DistillationCode0
Knowledge Distillation for End-to-End Person SearchCode0
KS-DETR: Knowledge Sharing in Attention Learning for Detection TransformerCode0
Cogni-Net: Cognitive Feature Learning through Deep Visual PerceptionCode0
Induced Model Matching: How Restricted Models Can Help Larger OnesCode0
Distilling Knowledge by Mimicking FeaturesCode0
InDistill: Information flow-preserving knowledge distillation for model compressionCode0
Induced Model Matching: Restricted Models Help Train Full-Featured ModelsCode0
Adversarial Teacher-Student Representation Learning for Domain GeneralizationCode0
UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit RepresentationCode0
Incremental Meta-Learning via Episodic Replay Distillation for Few-Shot Image RecognitionCode0
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech RecognitionCode0
Improving Stance Detection with Multi-Dataset Learning and Knowledge DistillationCode0
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from EnsemblesCode0
Improving Neural Topic Models with Wasserstein Knowledge DistillationCode0
Improving Question Answering Performance Using Knowledge Distillation and Active LearningCode0
Incorporating Graph Information in Transformer-based AMR ParsingCode0
EaSyGuide : ESG Issue Identification Framework leveraging Abilities of Generative Large Language ModelsCode0
CL-XABSA: Contrastive Learning for Cross-lingual Aspect-based Sentiment AnalysisCode0
Show:102550
← PrevPage 26 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified