SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 751–800 of 4240 papers

TitleStatusHype
Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?Code1
Distilling a Powerful Student Model via Online Knowledge DistillationCode1
Lightweight Transformers for Clinical Natural Language ProcessingCode1
A Discrepancy Aware Framework for Robust Anomaly DetectionCode1
Distilled Semantics for Comprehensive Scene Understanding from VideosCode1
Distilling Audio-Visual Knowledge by Compositional Contrastive LearningCode1
Camera clustering for scalable stream-based active distillationCode1
Contrastive Deep SupervisionCode1
Anomaly Detection in Video via Self-Supervised and Multi-Task LearningCode1
Decomposed Knowledge Distillation for Class-Incremental Semantic SegmentationCode1
DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge DistillationCode1
Contrastive Model Inversion for Data-Free Knowledge DistillationCode1
Contrastive Representation DistillationCode1
AutoGAN-Distiller: Searching to Compress Generative Adversarial NetworksCode1
Distilling Autoregressive Models to Obtain High-Performance Non-Autoregressive Solvers for Vehicle Routing Problems with Faster Inference SpeedCode1
LQER: Low-Rank Quantization Error Reconstruction for LLMsCode1
Distilling Object Detectors via Decoupled FeaturesCode1
Deep Graph-level Anomaly Detection by Glocal Knowledge DistillationCode1
DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue GenerationCode1
Mask-invariant Face Recognition through Template-level Knowledge DistillationCode1
mCLIP: Multilingual CLIP via Cross-lingual TransferCode1
MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge DistillationCode1
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without TricksCode1
ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot LearningCode1
Distillation and Refinement of Reasoning in Small Language Models for Document Re-rankingCode1
Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-ExpertsCode1
Meta-Learning based Degradation Representation for Blind Super-ResolutionCode1
BERT Learns to Teach: Knowledge Distillation with Meta LearningCode1
CaMEL: Mean Teacher Learning for Image CaptioningCode1
MetricGAN-OKD: Multi-Metric Optimization of MetricGAN via Online Knowledge Distillation for Speech EnhancementCode1
Distillation-Based Training for Multi-Exit ArchitecturesCode1
CaKDP: Category-aware Knowledge Distillation and Pruning Framework for Lightweight 3D Object DetectionCode1
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterCode1
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation TrainerCode1
MixSKD: Self-Knowledge Distillation from Mixup for Image RecognitionCode1
ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning ModelsCode1
Distilling Knowledge from Graph Convolutional NetworksCode1
MobileIQA: Exploiting Mobile-level Diverse Opinion Network For No-Reference Image Quality Assessment Using Knowledge DistillationCode1
A Knowledge Distillation Framework For Enhancing Ear-EEG Based Sleep Staging With Scalp-EEG DataCode1
Modality-Balanced Learning for Multimedia RecommendationCode1
MonoSKD: General Distillation Framework for Monocular 3D Object Detection via Spearman Correlation CoefficientCode1
MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object DetectionCode1
Mosaicking to Distill: Knowledge Distillation from Out-of-Domain DataCode1
MPCFormer: fast, performant and private Transformer inference with MPCCode1
DistilCSE: Effective Knowledge Distillation For Contrastive Sentence EmbeddingsCode1
Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal HashingCode1
Multi-Label Knowledge DistillationCode1
Multi-Level Branched Regularization for Federated LearningCode1
Multimodal and multiview distillation for real-time player detection on a football fieldCode1
DisCo: Distilled Student Models Co-training for Semi-supervised Text MiningCode1
Show:102550
← PrevPage 16 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified