SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3751–3800 of 4240 papers

TitleStatusHype
ReffAKD: Resource-efficient Autoencoder-based Knowledge DistillationCode0
Collaborative Learning of Bidirectional Decoders for Unsupervised Text Style TransferCode0
Refined Response Distillation for Class-Incremental Player DetectionCode0
MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face ImagesCode0
Image Recognition with Online Lightweight Vision Transformer: A SurveyCode0
Distilling Object Detectors With Global KnowledgeCode0
Low-Energy On-Device Personalization for MCUsCode0
MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLUCode0
Hybrid Data-Free Knowledge DistillationCode0
Collaborative Deep Reinforcement LearningCode0
Cogni-Net: Cognitive Feature Learning through Deep Visual PerceptionCode0
MimicGait: A Model Agnostic approach for Occluded Gait Recognition using Correlational Knowledge DistillationCode0
Regression-Oriented Knowledge Distillation for Lightweight Ship Orientation Angle Prediction with Optical Remote Sensing ImagesCode0
Distilling Object Detectors with Fine-grained Feature ImitationCode0
Hybrid Attention Model Using Feature Decomposition and Knowledge Distillation for Glucose ForecastingCode0
REHRSeg: Unleashing the Power of Self-Supervised Super-Resolution for Resource-Efficient 3D MRI SegmentationCode0
HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View DistillationCode0
Distilling Reasoning Capabilities into Smaller Language ModelsCode0
Minimizing PLM-Based Few-Shot Intent DetectorsCode0
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic SegmentationCode0
TSPipe: Learn from Teacher Faster with PipelinesCode0
Reinforced Knowledge Distillation for Time Series RegressionCode0
A Flexible Multi-Task Model for BERT ServingCode0
HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge DistillationCode0
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive TranslationCode0
Relational Diffusion Distillation for Efficient Image GenerationCode0
Relational Knowledge DistillationCode0
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model CompressionCode0
Distilling Model KnowledgeCode0
How to Train the Teacher Model for Effective Knowledge DistillationCode0
MixedTeacher : Knowledge Distillation for fast inference textural anomaly detectionCode0
Topology-Guided Knowledge Distillation for Efficient Point Cloud ProcessingCode0
Distilling Local Texture Features for Colorectal Tissue Classification in Low Data RegimesCode0
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning StrategyCode0
CL-XABSA: Contrastive Learning for Cross-lingual Aspect-based Sentiment AnalysisCode0
Mixture of Modular Experts: Distilling Knowledge from a Multilingual Teacher into Specialized Modular Language ModelsCode0
Relative Difficulty Distillation for Semantic SegmentationCode0
Self-Supervised Z-Slice Augmentation for 3D Bio-Imaging via Knowledge DistillationCode0
How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face RecognitionCode0
Releasing Graph Neural Networks with Differential Privacy GuaranteesCode0
Holistic White-light Polyp Classification via Alignment-free Dense Distillation of Auxiliary Optical ChromoendoscopyCode0
Distilling Knowledge for Empathy DetectionCode0
RELIANT: Fair Knowledge Distillation for Graph Neural NetworksCode0
HiTSR: A Hierarchical Transformer for Reference-based Super-ResolutionCode0
Highlight Every Step: Knowledge Distillation via Collaborative TeachingCode0
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image ClassificationCode0
Distilling Knowledge for Designing Computational Imaging SystemsCode0
MOD: A Deep Mixture Model with Online Knowledge Distillation for Large Scale Video Temporal Concept LocalizationCode0
Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and FusionCode0
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns ClusteringCode0
Show:102550
← PrevPage 76 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified