SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3201–3250 of 4240 papers

TitleStatusHype
Better Supervisory Signals by Observing Learning PathsCode0
MIAShield: Defending Membership Inference Attacks via Preemptive Exclusion of Members—0
Dual Embodied-Symbolic Concept Representations for Deep Learning—0
TRILLsson: Distilled Universal Paralinguistic Speech Representations—0
Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation—0
Joint Answering and Explanation for Visual Commonsense ReasoningCode0
Learn From the Past: Experience Ensemble Knowledge Distillation—0
Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation—0
Efficient Video Segmentation Models with Per-frame Inference—0
Are All Linear Regions Created Equal?Code0
Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification—0
Distilled Neural Networks for Efficient Learning to RankCode0
Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning—0
A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation—0
Cross-Task Knowledge Distillation in Multi-Task Recommendation—0
Knowledge Distillation with Deep SupervisionCode0
No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices—0
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation—0
Meta Knowledge Distillation—0
Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Search—0
AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation—0
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning—0
Locally Differentially Private Distributed Deep Learning via Knowledge DistillationCode0
Adaptive Mixing of Auxiliary Losses in Supervised LearningCode0
Measuring and Reducing Model Update Regression in Structured Prediction for NLP—0
Cross domain knowledge compression in realtime optical flow prediction on ultrasound sequences—0
Iterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition—0
Bootstrapped Representation Learning for Skeleton-Based Action Recognition—0
Deep-Disaster: Unsupervised Disaster Detection and Localization Using Visual DataCode0
Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning—0
Improving Robustness by Enhancing Weak SubnetsCode0
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models—0
Dynamic Rectification Knowledge DistillationCode0
Adaptive Instance Distillation for Object Detection in Autonomous Driving—0
TrustAL: Trustworthy Active Learning using Knowledge Distillation—0
One Student Knows All Experts Know: From Sparse to Dense—0
Jointly Learning Knowledge Embedding and Neighborhood Consensus with Relational Knowledge Distillation for Entity Alignment—0
Attentive Task Interaction Network for Multi-Task LearningCode0
Federated Unlearning with Knowledge Distillation—0
Can Model Compression Improve NLP Fairness—0
AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models—0
Image-to-Video Re-Identification via Mutual Discriminative Knowledge Transfer—0
UKD: Debiasing Conversion Rate Estimation via Uncertainty-regularized Knowledge Distillation—0
Improving Neural Machine Translation by Denoising Training—0
Continual Coarse-to-Fine Domain Adaptation in Semantic SegmentationCode0
Cross-modal Contrastive Distillation for Instructional Activity Anticipation—0
Knowledge Distillation as Self-Supervised Learning—0
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation—0
Re2G: Retrieve, Rerank, Generate—0
Learning Cross-Lingual IR from an English Retriever—0
Show:102550
← PrevPage 65 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified