SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2401–2450 of 4240 papers

TitleStatusHype
Population-Based Evolutionary Gaming for Unsupervised Person Re-identification—0
Regularized Evolutionary Population-Based Training—0
Pose-Guided Feature Learning with Knowledge Distillation for Occluded Person Re-Identification—0
Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer—0
Positive-Unlabeled Data Purification in the Wild for Object Detection—0
Poster: Self-Supervised Quantization-Aware Knowledge Distillation—0
PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation—0
PP-StructureV2: A Stronger Document Analysis System—0
PQDAST: Depth-Aware Arbitrary Style Transfer for Games via Perceptual Quality-Guided Distillation—0
PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation—0
Practical Insights into Knowledge Distillation for Pre-Trained Models—0
Practical Knowledge Distillation: Using DNNs to Beat DNNs—0
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation—0
Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation—0
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison—0
Preserving Node Distinctness in Graph Autoencoders via Similarity Distillation—0
Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation—0
Pre-trained Language Model and Knowledge Distillation for Lightweight Sequential Recommendation—0
Pre-trained Model Guided Mixture Knowledge Distillation for Adversarial Federated Learning—0
Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis—0
Pre-Trained Vision-Language Models as Partial Annotators—0
Pre-training Distillation for Large Language Models: A Design Space Exploration—0
Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG—0
Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data—0
Preventing Distillation-based Attacks on Neural Network IP—0
Preview-based Category Contrastive Learning for Knowledge Distillation—0
Prime-Aware Adaptive Distillation—0
Prior knowledge distillation based on financial time series—0
Prior Knowledge Distillation Network for Face Super-Resolution—0
Prior Knowledge Guided Network for Video Anomaly Detection—0
Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models—0
Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator—0
Privacy-preserving Fine-tuning of Large Language Models through Flatness—0
Private Deep Learning with Teacher Ensembles—0
Private Model Compression via Knowledge Distillation—0
Privileged Knowledge Distillation for Online Action Detection—0
Proactive Detection and Calibration of Seasonal Advertisements with Multimodal Large Language Models—0
Proactive Guidance of Multi-Turn Conversation in Industrial Search—0
Proactive Sequence Generator via Knowledge Acquisition—0
Probabilistic Integration of Object Level Annotations in Chest X-ray Classification—0
Probabilistic Knowledge Distillation of Face Ensembles—0
Probabilistic Self-supervised Learning via Scoring Rules Minimization—0
PROD: Progressive Distillation for Dense Retrieval—0
ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes—0
Progressive Class-level Distillation—0
Progressive Collaborative and Semantic Knowledge Fusion for Generative Recommendation—0
Progressive Cross-modal Knowledge Distillation for Human Action Recognition—0
Progressive distillation induces an implicit curriculum—0
Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks—0
ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition—0
Show:102550
← PrevPage 49 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified