SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3951–4000 of 4240 papers

TitleStatusHype
Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor AttacksCode0
Students are the Best Teacher: Exit-Ensemble Distillation with Multi-ExitsCode0
Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning StrategiesCode0
NC-NCD: Novel Class Discovery for Node ClassificationCode0
Robust Model Compression Using Deep HypothesesCode0
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and TasksCode0
Neighborhood Commonality-aware Evolution Network for Continuous Generalized Category DiscoveryCode0
Robust Multimodal Segmentation with Representation Regularization and Hybrid Prototype DistillationCode0
Robustness and Diversity Seeking Data-Free Knowledge DistillationCode0
FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question AnsweringCode0
Towards Disturbance-Free Visual Mobile ManipulationCode0
Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge IntegrationCode0
Adversarial Teacher-Student Representation Learning for Domain GeneralizationCode0
Network Pruning via Transformable Architecture SearchCode0
Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion AugmentationCode0
Differentially Private Knowledge Distillation via Synthetic Text GenerationCode0
Detect, Distill and Update: Learned DB Systems Facing Out of Distribution DataCode0
ROD: Reception-aware Online Distillation for Sparse GraphsCode0
Detect, Distill and Update: Detect, Distill and Update: Learned DB Systems Facing Out of Distribution DataCode0
Neural Network Pruning with Residual-Connections and Limited-DataCode0
FlowDistill: Scalable Traffic Flow Prediction via Distillation from LLMsCode0
Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental LearningCode0
AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge DistillationCode0
New Insights on Relieving Task-Recency Bias for Online Class Incremental LearningCode0
Dense 2D-3D Indoor Prediction with Sound via Aligned Cross-Modal DistillationCode0
ROSAR: An Adversarial Re-Training Framework for Robust Side-Scan Sonar Object DetectionCode0
Adversarial Moment-Matching Distillation of Large Language ModelsCode0
Delta Distillation for Efficient Video ProcessingCode0
AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained ModelsCode0
Few Shot Network Compression via Cross DistillationCode0
Sub-goal Distillation: A Method to Improve Small Language AgentsCode0
Few-shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge DistillationCode0
RUIE: Retrieval-based Unified Information Extraction using Large Language ModelCode0
deepQuest-py: Large and Distilled Models for Quality EstimationCode0
WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary KnowledgeCode0
Subspace Distillation for Continual LearningCode0
A Forward and Backward Compatible Framework for Few-shot Class-incremental Pill RecognitionCode0
"No Matter What You Do": Purifying GNN Models via Backdoor UnlearningCode0
Integrating Translation Memories into Non-Autoregressive Machine TranslationCode0
Non-Autoregressive Neural Machine TranslationCode0
Uncertainty-Guided Cross Attention Ensemble Mean Teacher for Semi-supervised Medical Image SegmentationCode0
CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature SharingCode0
Knowledge Distillation with Deep SupervisionCode0
Catch-Up Distillation: You Only Need to Train Once for Accelerating SamplingCode0
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free ReplayCode0
FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated LearningCode0
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge DistillationCode0
Exploring Vacant Classes in Label-Skewed Federated LearningCode0
SaiT: Sparse Vision Transformers through Adaptive Token PruningCode0
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxCode0
Show:102550
← PrevPage 80 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified