SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3701–3750 of 4240 papers

TitleStatusHype
Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer—0
DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling—0
Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models—0
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge DistillationCode1
Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review—0
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate LayersCode0
A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions—0
Improving Neural Topic Models using Knowledge DistillationCode1
Self-training Improves Pre-training for Natural Language UnderstandingCode1
Lifelong Language Knowledge DistillationCode1
Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation—0
Neighbourhood Distillation: On the benefits of non end-to-end distillation—0
Online Knowledge Distillation via Multi-branch Diversity Enhancement—0
WeChat Neural Machine Translation Systems for WMT20—0
Improved Knowledge Distillation via Full Kernel Matrix TransferCode0
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks—0
Pea-KD: Parameter-efficient and Accurate Knowledge Distillation on BERT—0
TinyGAN: Distilling BigGAN for Conditional Image GenerationCode1
Contrastive Distillation on Intermediate Representations for Language Model CompressionCode1
Pea-KD: Parameter-efficient and accurate Knowledge Distillation—0
Kernel Based Progressive Distillation for Adder Neural Networks—0
Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach—0
TernaryBERT: Distillation-aware Ultra-low Bit BERTCode0
N-LTP: An Open-source Neural Language Technology Platform for ChineseCode3
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey—0
Multi-Frame to Single-Frame: Knowledge Distillation for 3D Object Detection—0
Open-set Short Utterance Forensic Speaker Verification using Teacher-Student Network with Explicit Inductive Bias—0
EI-MTD:Moving Target Defense for Edge Intelligence against Adversarial Attacks—0
Weight Distillation: Transferring the Knowledge in Neural Network Parameters—0
Introspective Learning by Distilling Knowledge from Online Self-explanation—0
Densely Guided Knowledge Distillation using Multiple Teacher AssistantsCode1
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning—0
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without TricksCode1
S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric LearningCode1
Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP—0
Simplified TinyBERT: Knowledge Distillation for Document RetrievalCode1
Noisy Self-Knowledge Distillation for Text SummarizationCode1
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition—0
Autoregressive Knowledge Distillation through Imitation LearningCode0
SSKD: Self-Supervised Knowledge Distillation for Cross Domain Adaptive Person Re-Identification—0
BoostingBERT:Integrating Multi-Class Boosting into BERT for NLP Tasks—0
DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning—0
Extending Label Smoothing Regularization with Self-Knowledge Distillation—0
On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective—0
Simulating Unknown Target Models for Query-Efficient Black-box AttacksCode1
SAIL: Self-Augmented Graph Contrastive Learning—0
Lifelong Object Detection—0
Classification of Diabetic Retinopathy Using Unlabeled Data and Knowledge Distillation—0
Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action RecognitionCode1
Automatic Assignment of Radiology Examination Protocols Using Pre-trained Language Models with Knowledge DistillationCode0
Show:102550
← PrevPage 75 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified