SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 501–550 of 4240 papers

TitleStatusHype
Learning Compatible EmbeddingsCode1
Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation ModelsCode1
Advantage-Guided Distillation for Preference Alignment in Small Language ModelsCode1
Dynamic Knowledge Distillation for Pre-trained Language ModelsCode1
Dynamic Temperature Knowledge DistillationCode1
Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed ClassificationCode1
A Deep Knowledge Distillation framework for EEG assisted enhancement of single-lead ECG based sleep stagingCode1
EasyST: A Simple Framework for Spatio-Temporal PredictionCode1
EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic DataCode1
APSNet: Attention Based Point Cloud SamplingCode1
Context-Aware Image Inpainting with Learned Semantic PriorsCode1
FAMIE: A Fast Active Learning Framework for Multilingual Information ExtractionCode1
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary CellsCode1
Consistent Representation Learning for Continual Relation ExtractionCode1
Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly DetectionCode1
ConStyle v2: A Strong Prompter for All-in-One Image RestorationCode1
Designing Large Foundation Models for Efficient Training and Inference: A SurveyCode1
Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy AnnotationsCode1
Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation FrameworkCode1
The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single ImageCode1
Conformer and Blind Noisy Students for Improved Image Quality AssessmentCode1
CoNMix for Source-free Single and Multi-target Domain AdaptationCode1
Continual Learning for Image Segmentation with Dynamic QueryCode1
ConNER: Consistency Training for Cross-lingual Named Entity RecognitionCode1
A Contrastive Distillation Approach for Incremental Semantic Segmentation in Aerial ImagesCode1
Content-Aware GAN CompressionCode1
Cross-Layer Distillation with Semantic CalibrationCode1
FairDistillation: Mitigating Stereotyping in Language ModelsCode1
FastSpeech 2: Fast and High-Quality End-to-End Text to SpeechCode1
FerKD: Surgical Label Adaptation for Efficient DistillationCode1
Computation-Efficient Knowledge Distillation via Uncertainty-Aware MixupCode1
Compressing Deep Graph Neural Networks via Adversarial Knowledge DistillationCode1
Exploring Extreme Parameter Compression for Pre-trained Language ModelsCode1
Comprehensive Knowledge Distillation with Causal InterventionCode1
A Dual-Space Framework for General Knowledge Distillation of Large Language ModelsCode1
ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian SplattingCode1
Exploring Inter-Channel Correlation for Diversity-Preserved Knowledge DistillationCode1
Complementary Relation Contrastive DistillationCode1
Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot LearningCode1
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge DistillationCode1
Exploring Deeper! Segment Anything Model with Depth Perception for Camouflaged Object DetectionCode1
Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationCode1
Anti-Distillation Backdoor Attacks: Backdoors Can Really Survive in Knowledge DistillationCode1
Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-DistillationCode1
Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side DistillationCode1
COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using TransformersCode1
Evolving Search Space for Neural Architecture SearchCode1
Collaborative Distillation for Ultra-Resolution Universal Style TransferCode1
AD-KD: Attribution-Driven Knowledge Distillation for Language Model CompressionCode1
Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error CorrectionCode1
Show:102550
← PrevPage 11 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified