SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3101–3150 of 4240 papers

TitleStatusHype
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding—0
Limitations of Knowledge Distillation for Zero-shot Transfer Learning—0
Distilling Object Detectors with Feature RichnessCode1
Learning Distilled Collaboration Graph for Multi-Agent PerceptionCode1
PP-ShiTu: A Practical Lightweight Image Recognition SystemCode0
Rethinking the Knowledge Distillation From the Perspective of Model Calibration—0
Estimating and Maximizing Mutual Information for Knowledge Distillation—0
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks—0
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM—0
Towards Model Agnostic Federated Learning Using Knowledge Distillation—0
Temporal Knowledge Distillation for On-device Audio Classification—0
GenURL: A General Framework for Unsupervised Representation Learning—0
Mosaicking to Distill: Knowledge Distillation from Out-of-Domain DataCode1
Beyond Classification: Knowledge Distillation using Multi-Object Impressions—0
Response-based Distillation for Incremental Object Detection—0
Instance-Conditional Knowledge Distillation for Object DetectionCode1
Reconstructing Pruned Filters using Cheap Spatial Transformations—0
MUSE: Feature Self-Distillation with Mutual Information and Self-Information—0
Anti-Distillation Backdoor Attacks: Backdoors Can Really Survive in Knowledge DistillationCode1
X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation—0
How and When Adversarial Robustness Transfers in Knowledge Distillation?—0
Pseudo Supervised Monocular Depth Estimation with Teacher-Student Network—0
Pixel-by-Pixel Cross-Domain Alignment for Few-Shot Semantic SegmentationCode1
Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression—0
Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach—0
Class Incremental Online Streaming Learning—0
FedHe: Heterogeneous Models and Communication-Efficient Federated LearningCode0
Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient DistillationCode0
Graph-less Neural Networks: Teaching Old MLPs New Tricks via DistillationCode1
Know your tools well: Better and faster QA with synthetic examples—0
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model CompressionCode0
A Short Study on Compressing Decoder-Based Language Models—0
Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding—0
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher—0
From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation—0
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm—0
Multilingual Neural Machine Translation:Can Linguistic Hierarchies Help?—0
Kronecker Decomposition for GPT Compression—0
FocusNet: Classifying Better by Focusing on Confusing ClassesCode1
Symbolic Knowledge Distillation: from General Language Models to Commonsense ModelsCode1
Language Modelling via Learning to Rank—0
False Negative Distillation and Contrastive Learning for Personalized Outfit Recommendation—0
Object DGCNN: 3D Object Detection using Dynamic GraphsCode1
CONetV2: Efficient Auto-Channel Size Optimization for CNNsCode0
Rectifying the Data Bias in Knowledge Distillation—0
Compact CNN Models for On-device Ocular-based User Recognition in Mobile Devices—0
Towards Streaming Egocentric Action Anticipation—0
Visualizing the embedding space to explain the effect of knowledge distillation—0
Towards Data-Free Domain GeneralizationCode0
Cross-modal Knowledge Distillation for Vision-to-Sensor Action RecognitionCode0
Show:102550
← PrevPage 63 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified