SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2451–2500 of 4240 papers

TitleStatusHype
Decomposed Knowledge Distillation for Class-Incremental Semantic SegmentationCode1
SaiT: Sparse Vision Transformers through Adaptive Token PruningCode0
Linkless Link Prediction via Relational Distillation—0
Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR—0
Detect, Distill and Update: Learned DB Systems Facing Out of Distribution DataCode0
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
APSNet: Attention Based Point Cloud SamplingCode1
PP-StructureV2: A Stronger Document Analysis System—0
Meta-Learning with Self-Improving Momentum TargetCode1
ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot LearningCode1
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes—0
Patch-based Knowledge Distillation for Lifelong Person Re-IdentificationCode1
Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again—0
Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine TranslationCode1
Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks—0
Let Images Give You More:Point Cloud Cross-Modal Training for Shape AnalysisCode2
Students taught by multimodal teachers are superior action recognizers—0
Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-ExpertsCode1
Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization—0
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video RetrievalCode1
Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image ClassificationCode1
IDa-Det: An Information Discrepancy-aware Distillation for 1-bit DetectorsCode1
CLIP model is an Efficient Continual LearnerCode1
Effective Self-supervised Pre-training on Low-compute Networks without DistillationCode1
AlphaFold Distillation for Protein DesignCode1
Meta-Ensemble Parameter Learning—0
Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation—0
Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning—0
Positive Pair Distillation Considered Harmful: Continual Meta Metric Learning for Lifelong Object Re-IdentificationCode0
Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Search—0
A Study on the Efficiency and Generalization of Light Hybrid Retrievers—0
Robust Active Distillation—0
Attention Distillation: self-supervised vision transformer students need more guidanceCode1
Knowledge Transfer with Visual Prompt in multi-modal Dialogue Understanding and Generation—0
Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model CompressionCode0
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation—0
TAKE: Topic-shift Aware Knowledge sElection for Dialogue GenerationCode0
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing—0
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment ClassificationCode0
Sentiment Interpretable Logic Tensor Network for Aspect-Term Sentiment Analysis—0
Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition—0
Multimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised LearningCode1
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language ModelsCode0
Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation—0
Slimmable Networks for Contrastive Self-supervised LearningCode0
Using Knowledge Distillation to improve interpretable models in a retail banking context—0
Towards a Unified View of Affinity-Based Knowledge Distillation—0
Label driven Knowledge Distillation for Federated Learning with non-IID Data—0
Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face RecognitionCode1
Hyper-Representations as Generative Models: Sampling Unseen Neural Network WeightsCode1
Show:102550
← PrevPage 50 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified