SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 701–750 of 4240 papers

TitleStatusHype
Symbolic Knowledge Distillation: from General Language Models to Commonsense ModelsCode1
FocusNet: Classifying Better by Focusing on Confusing ClassesCode1
Object DGCNN: 3D Object Detection using Dynamic GraphsCode1
Towards Accurate Cross-Domain In-Bed Human Pose EstimationCode1
KNOT: Knowledge Distillation using Optimal Transport for Solving NLP TasksCode1
Prune Your Model Before Distill ItCode1
Multilingual AMR Parsing with Noisy Knowledge DistillationCode1
Deep Structured Instance Graph for Distilling Object DetectorsCode1
Dynamic Knowledge Distillation for Pre-trained Language ModelsCode1
Segmentation with mixed supervision: Confidence maximization helps knowledge distillationCode1
Distilling Linguistic Context for Language Model CompressionCode1
The NiuTrans System for the WMT21 Efficiency TaskCode1
The NiuTrans System for WNGT 2020 Efficiency TaskCode1
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge DistillationCode1
Multi-Scale Aligned Distillation for Low-Resolution DetectionCode1
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language UnderstandingCode1
Knowledge Distillation Using Hierarchical Self-Supervision Augmented DistributionCode1
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT CompressionCode1
Black-Box Attacks on Sequential Recommenders via Data-Free Model ExtractionCode1
Cross-category Video Highlight Detection via Set-based LearningCode1
PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge DistillationCode1
Efficient Medical Image Segmentation Based on Knowledge DistillationCode1
Supervised Compression for Resource-Constrained Edge Computing SystemsCode1
Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality AssessmentCode1
Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student BetterCode1
AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric LearningCode1
Distilling Holistic Knowledge with Graph Neural NetworksCode1
Transferring Knowledge Distillation for Multilingual Social Event DetectionCode1
Knowledge Distillation from BERT Transformer to Speech Transformer for Intent ClassificationCode1
Learning Compatible EmbeddingsCode1
Online Knowledge Distillation for Efficient Pose EstimationCode1
Hierarchical Self-supervised Augmented Knowledge DistillationCode1
Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy AnnotationsCode1
Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image ClassificationCode1
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge TransferCode1
Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural NetworkCode1
Learning Efficient Vision Transformers via Fine-Grained Manifold DistillationCode1
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and RetrievalCode1
SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental LearningCode1
Structured Sparse R-CNN for Direct Scene Graph GenerationCode1
Context-Aware Image Inpainting with Learned Semantic PriorsCode1
Does Knowledge Distillation Really Work?Code1
Distilling Image Classifiers in Object DetectorsCode1
BERT Learns to Teach: Knowledge Distillation with Meta LearningCode1
XtremeDistilTransformers: Task Transfer for Task-agnostic DistillationCode1
Zero-Shot Knowledge Distillation from a Decision-Based Black-Box ModelCode1
Preservation of the Global Knowledge by Not-True Distillation in Federated LearningCode1
Bidirectional Distillation for Top-K Recommender SystemCode1
Towards Quantifiable Dialogue Coherence EvaluationCode1
Transformer-Based Source-Free Domain AdaptationCode1
Show:102550
← PrevPage 15 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified