SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 851–900 of 4240 papers

TitleStatusHype
Simplified TinyBERT: Knowledge Distillation for Document RetrievalCode1
Noisy Self-Knowledge Distillation for Text SummarizationCode1
Simulating Unknown Target Models for Query-Efficient Black-box AttacksCode1
Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action RecognitionCode1
Unpaired Learning of Deep Image DenoisingCode1
Performance Optimization for Federated Person Re-identification via Benchmark AnalysisCode1
PARADE: Passage Representation Aggregation for Document RerankingCode1
Knowledge Transfer via Dense Cross-Layer Mutual-DistillationCode1
Distilling the Knowledge of BERT for Sequence-to-Sequence ASRCode1
Improving Knowledge Distillation via Category StructureCode1
Distilling Visual Priors from Self-Supervised LearningCode1
Intra-class Feature Variation Distillation for Semantic SegmentationCode1
Weakly Supervised 3D Object Detection from Point CloudsCode1
Group Knowledge Transfer: Federated Learning of Large CNNs at the EdgeCode1
Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance SegmentationCode1
Resolution Switchable Networks for Runtime Efficient Image RecognitionCode1
Self-supervision on Unlabelled OR Data for Multi-person 2D/3D Human Pose EstimationCode1
Defocus Blur Detection via Depth DistillationCode1
Knowledge Distillation for Multi-task LearningCode1
Unsupervised Multi-Target Domain Adaptation Through Knowledge DistillationCode1
RATT: Recurrent Attention to Transient Tasks for Continual Image CaptioningCode1
Towards Practical Lipreading with Distilled and Efficient ModelsCode1
Learning to Learn Parameterized Classification Networks for Scalable Input ImagesCode1
Temporal Self-Ensembling Teacher for Semi-Supervised Object DetectionCode1
Robust Re-Identification by Multiple Views Knowledge DistillationCode1
Tracking-by-Trackers with a Distilled and Reinforced ModelCode1
Improving Weakly Supervised Visual Grounding by Contrastive Knowledge DistillationCode1
Improving Event Detection via Open-domain Trigger KnowledgeCode1
Self-Knowledge Distillation with Progressive Refinement of TargetsCode1
Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble DistillationCode1
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine TranslationCode1
Self-supervised Knowledge Distillation for Few-shot LearningCode1
AutoGAN-Distiller: Searching to Compress Generative Adversarial NetworksCode1
Knowledge Distillation Meets Self-SupervisionCode1
Real-Time Video Inference on Edge Devices via Adaptive Model StreamingCode1
Adjoined Networks: A Training Paradigm with Applications to Network CompressionCode1
FastSpeech 2: Fast and High-Quality End-to-End Text to SpeechCode1
Multi-view Contrastive Learning for Online Knowledge DistillationCode1
Peer Collaborative Learning for Online Knowledge DistillationCode1
Channel Distillation: Channel-Wise Attention for Knowledge DistillationCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Distilling Cross-Task Knowledge via Relationship MatchingCode1
Online Knowledge Distillation via Collaborative LearningCode1
Transferring Inductive Biases through Knowledge DistillationCode1
Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech RecognitionCode1
MicroNet for Efficient Language ModelingCode1
Data-Free Network Quantization With Adversarial Knowledge DistillationCode1
ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural NetworksCode1
MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient EstimationCode1
Heterogeneous Knowledge Distillation using Information Flow ModelingCode1
Show:102550
← PrevPage 18 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified