SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3251–3300 of 4240 papers

TitleStatusHype
Multi-granularity for knowledge distillationCode0
Online Continual Learning For Visual Food Classification—0
AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric LearningCode1
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval—0
Learning from Matured Dumb Teacher for Fine Generalization—0
Distilling Holistic Knowledge with Graph Neural NetworksCode1
Semi-Supervised Domain Generalizable Person Re-IdentificationCode2
Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data—0
Lifelong Intent Detection via Multi-Strategy Rebalancing—0
Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection—0
A distillation based approach for the diagnosis of diseases—0
Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading—0
Transferring Knowledge Distillation for Multilingual Social Event DetectionCode1
Decoupled Transformer for Scalable Inference in Open-domain Question Answering—0
Knowledge Distillation from BERT Transformer to Speech Transformer for Intent ClassificationCode1
WeChat Neural Machine Translation Systems for WMT21—0
MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets—0
Online Knowledge Distillation for Efficient Pose EstimationCode1
Learning Compatible EmbeddingsCode1
Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR—0
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval—0
On Knowledge Distillation for Translating Erroneous Speech Transcriptions—0
NAIST English-to-Japanese Simultaneous Translation System for IWSLT 2021 Simultaneous Text-to-text Task—0
The USYD-JD Speech Translation System for IWSLT2021—0
基于层间知识蒸馏的神经机器翻译(Inter-layer Knowledge Distillation for Neural Machine Translation)—0
Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension—0
Samsung R&D Institute Poland submission to WAT 2021 Indic Language Multilingual Task—0
POS-Constrained Parallel Decoding for Non-autoregressive GenerationCode0
Trigger is Not Sufficient: Exploiting Frame-aware Knowledge for Implicit Event Argument Extraction—0
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation—0
Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation—0
Pose-Guided Feature Learning with Knowledge Distillation for Occluded Person Re-Identification—0
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation SignalsCode0
QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning—0
Hierarchical Self-supervised Augmented Knowledge DistillationCode1
Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation—0
In Defense of the Learning Without Forgetting for Task Incremental Learning—0
Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation—0
IE-GAN: An Improved Evolutionary Generative Adversarial Network Using a New Fitness Function and a Generic Crossover OperatorCode0
ROD: Reception-aware Online Distillation for Sparse GraphsCode0
The USYD-JD Speech Translation System for IWSLT 2021—0
Follow Your Path: a Progressive Method for Knowledge Distillation—0
Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System—0
Double Similarity Distillation for Semantic Image Segmentation—0
Federated Action Recognition on Heterogeneous Embedded Devices—0
Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search—0
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task—0
A Flexible Multi-Task Model for BERT ServingCode0
Technical Report of Team GraphMIRAcles in the WikiKG90M-LSC Track of OGB-LSC @ KDD Cup 2021—0
Contrast R-CNN for Continual Learning in Object Detection—0
Show:102550
← PrevPage 66 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified