SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1201–1250 of 4240 papers

TitleStatusHype
D3T-GAN: Data-Dependent Domain Transfer GANs for Few-shot Image Generation—0
Aligning Teacher with Student Preferences for Tailored Training Data Generation—0
D^3ETR: Decoder Distillation for Detection Transformer—0
BEExformer: A Fast Inferencing Transformer Architecture via Binarization with Multiple Early Exits—0
Estimating Human Poses Across Datasets: A Unified Skeleton and Multi-Teacher Distillation Approach—0
Adaptive Explicit Knowledge Transfer for Knowledge Distillation—0
CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation—0
VRM: Knowledge Distillation via Virtual Relation Matching—0
Evaluation-oriented Knowledge Distillation for Deep Face Recognition—0
Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures—0
BD-KD: Balancing the Divergences for Online Knowledge Distillation—0
Essence Knowledge Distillation for Speech Recognition—0
CULL-MT: Compression Using Language and Layer pruning for Machine Translation—0
CTC Blank Triggered Dynamic Layer-Skipping for Efficient CTC-based Speech Recognition—0
Bayesian-Optimized One-Step Diffusion Model with Knowledge Distillation for Real-Time 3D Human Motion Prediction—0
Aligned Weight Regularizers for Pruning Pretrained Neural Networks—0
Accurate and Structured Pruning for Efficient Automatic Speech Recognition—0
Estimating and Maximizing Mutual Information for Knowledge Distillation—0
Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillation—0
Examining the Mapping Functions of Denoising Autoencoders in Singing Voice Separation—0
Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation—0
Crowd Counting with Online Knowledge Learning—0
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization—0
Cross-Task Knowledge Distillation in Multi-Task Recommendation—0
BanglaEmbed: Efficient Sentence Embedding Models for a Low-Resource Language Using Cross-Lingual Distillation Techniques—0
Error Exponent in Agnostic PAC Learning—0
Canine EEG Helps Human: Cross-Species and Cross-Modality Epileptic Seizure Detection via Multi-Space Alignment—0
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation—0
AlignCap: Aligning Speech Emotion Captioning to Human Preferences—0
Crossmodal Knowledge Distillation with WordNet-Relaxed Text Embeddings for Robust Image Classification—0
A predictive machine learning force field framework for liquid electrolyte development—0
Adaptive Deep Iris Feature Extractor at Arbitrary Resolutions—0
ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval—0
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining—0
Cross-modal knowledge distillation for action recognition—0
Balancing Cost and Benefit with Tied-Multi Transformers—0
Balanced softmax cross-entropy for incremental learning with and without memory—0
AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate—0
ATLAS: Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data—0
Cross Modal Distillation for Flood Extent Mapping—0
Cross-modal Contrastive Distillation for Instructional Activity Anticipation—0
Balance Divergence for Knowledge Distillation—0
Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages—0
Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation—0
Baidu Neural Machine Translation Systems for WMT19—0
Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech—0
Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages—0
Knowledge Distillation for Human Action Anticipation—0
Adaptive Beam Search to Enhance On-device Abstractive Summarization—0
Cross-Level Multi-Instance Distillation for Self-Supervised Fine-Grained Visual Categorization—0
Show:102550
← PrevPage 25 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified