SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 4101–4150 of 4240 papers

TitleStatusHype
Attend, Distill, Detect: Attention-aware Entropy Distillation for Anomaly DetectionCode0
FAKD: Feature Augmented Knowledge Distillation for Semantic SegmentationCode0
On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small ModelsCode0
On the Transferability of Visual Features in Generalized Zero-Shot LearningCode0
A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-DistillationCode0
On the Use of External Data for Spoken Named Entity RecognitionCode0
OpenGrok: Enhancing SNS Data Processing with Distilled Knowledge and Mask-like MechanismsCode0
Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep NetworksCode0
Born Again Neural NetworksCode0
Data-Free Knowledge Distillation for Image Super-ResolutionCode0
Faithful Label-free Knowledge DistillationCode0
Data-free Knowledge Distillation for Fine-grained Visual CategorizationCode0
Self-Attentive Spatio-Temporal Calibration for Precise Intermediate Layer Matching in ANN-to-SNN DistillationCode0
Fairness without Demographics through Knowledge DistillationCode0
Towards Real-time Video Compressive Sensing on Mobile DevicesCode0
Boosting Summarization with Normalizing Flows and Aggressive TrainingCode0
Self-Distillation for Gaussian Process Regression and ClassificationCode0
Data-free Knowledge Distillation for Segmentation using Data-Enriching GANCode0
Optimal Transport Guided Correlation Assignment for Multimodal Entity LinkingCode0
Teacher Agent: A Knowledge Distillation-Free Framework for Rehearsal-based Video Incremental LearningCode0
Facilitating Pornographic Text Detection for Open-Domain Dialogue Systems via Knowledge Distillation of Large Language ModelsCode0
Facilitating NSFW Text Detection in Open-Domain Dialogue Systems via Knowledge DistillationCode0
Optimizing edge AI models on HPC systems with the edge in the loopCode0
Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge DistillationCode0
Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision TransformersCode0
Teacher Network Calibration Improves Cross-Quality Knowledge DistillationCode0
Facial Landmark Points Detection Using Knowledge Distillation-Based Neural NetworksCode0
Boosting Residual Networks with Group KnowledgeCode0
Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual LearningCode0
Model-Based Reinforcement Learning with Multi-Task Offline PretrainingCode0
When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?Code0
Eyelid’s Intrinsic Motion-aware Feature Learning for Real-time Eyeblink Detection in the WildCode0
An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language UnderstandingCode0
ORC: Network Group-based Knowledge Distillation using Online Role ChangeCode0
Data-Free Generative Replay for Class-Incremental Learning on Imbalanced DataCode0
Exploring Target Representations for Masked AutoencodersCode0
Adaptive Mixing of Auxiliary Losses in Supervised LearningCode0
Distilling the Unknown to Unveil CertaintyCode0
Exploring Social Media for Early Detection of Depression in COVID-19 PatientsCode0
Exploring Non-Autoregressive Text Style TransferCode0
Exploring Hyperspectral Anomaly Detection with Human Vision: A Small Target Aware DetectorCode0
Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency PerspectiveCode0
A Diversity-Enhanced Knowledge Distillation Model for Practical Math Word Problem SolvingCode0
Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge DistillationCode0
Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge DistillationCode0
Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple LevelsCode0
OVOSE: Open-Vocabulary Semantic Segmentation in Event-Based CamerasCode0
Evolutionary Generative Adversarial Networks with Crossover Based Knowledge DistillationCode0
PruMUX: Augmenting Data Multiplexing with Model CompressionCode0
Weak-to-Strong 3D Object Detection with X-Ray DistillationCode0
Show:102550
← PrevPage 83 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified