SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 801–850 of 4240 papers

TitleStatusHype
CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray SegmentationCode1
Show, Attend and Distill:Knowledge Distillation via Attention-based Feature MatchingCode1
ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning ModelsCode1
Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff PerspectiveCode1
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay BufferCode1
SEED: Self-supervised Distillation For Visual RepresentationCode1
Knowledge Distillation in Iterative Generative Models for Improved Sampling SpeedCode1
Self-Mutual Distillation Learning for Continuous Sign Language RecognitionCode1
Exploring Inter-Channel Correlation for Diversity-Preserved Knowledge DistillationCode1
Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient DetectorsCode1
Unified Mandarin TTS Front-end Based on Distilled BERT ModelCode1
CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models CascadeCode1
Learning Light-Weight Translation Models from Deep TransformerCode1
Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose EstimationCode1
Computation-Efficient Knowledge Distillation via Uncertainty-Aware MixupCode1
Progressive Network Grafting for Few-Shot Knowledge DistillationCode1
Distilling Knowledge from Reader to Retriever for Question AnsweringCode1
DE-RRD: A Knowledge Distillation Framework for Recommender SystemCode1
Cross-Layer Distillation with Semantic CalibrationCode1
What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical PerspectiveCode1
Going Beyond Classification Accuracy Metrics in Model CompressionCode1
Multi-level Knowledge Distillation via Knowledge Alignment and CorrelationCode1
Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient SpaceCode1
Knowledge Base Embedding By Cooperative Knowledge DistillationCode1
Task-Oriented Feature DistillationCode1
KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and QuantizationCode1
Prototype-based Incremental Few-Shot Semantic SegmentationCode1
Channel-wise Knowledge Distillation for Dense PredictionCode1
Multiresolution Knowledge Distillation for Anomaly DetectionCode1
Evolving Search Space for Neural Architecture SearchCode1
Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing SystemsCode1
KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge DistillationCode1
Anomaly Detection in Video via Self-Supervised and Multi-Task LearningCode1
Federated Knowledge DistillationCode1
Domain Adaptive Knowledge Distillation for Driving Scene Semantic SegmentationCode1
FastFormers: Highly Efficient Transformer Models for Natural Language UnderstandingCode1
Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine TranslationCode1
Distilling Dense Representations for Ranking using Tightly-Coupled TeachersCode1
Knowledge Distillation for BERT Unsupervised Domain AdaptationCode1
Reducing the Teacher-Student Gap via Spherical Knowledge DisitllationCode1
Task Decoupled Knowledge Distillation For Lightweight Face DetectorsCode1
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge DistillationCode1
Improving Neural Topic Models using Knowledge DistillationCode1
Lifelong Language Knowledge DistillationCode1
Self-training Improves Pre-training for Natural Language UnderstandingCode1
Contrastive Distillation on Intermediate Representations for Language Model CompressionCode1
TinyGAN: Distilling BigGAN for Conditional Image GenerationCode1
Densely Guided Knowledge Distillation using Multiple Teacher AssistantsCode1
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without TricksCode1
S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric LearningCode1
Show:102550
← PrevPage 17 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified