SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3501–3525 of 4240 papers

TitleStatusHype
Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection—0
A distillation based approach for the diagnosis of diseases—0
Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading—0
Decoupled Transformer for Scalable Inference in Open-domain Question Answering—0
MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets—0
WeChat Neural Machine Translation Systems for WMT21—0
Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR—0
On Knowledge Distillation for Translating Erroneous Speech Transcriptions—0
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval—0
Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension—0
Samsung R&D Institute Poland submission to WAT 2021 Indic Language Multilingual Task—0
The USYD-JD Speech Translation System for IWSLT2021—0
NAIST English-to-Japanese Simultaneous Translation System for IWSLT 2021 Simultaneous Text-to-text Task—0
Trigger is Not Sufficient: Exploiting Frame-aware Knowledge for Implicit Event Argument Extraction—0
基于层间知识蒸馏的神经机器翻译(Inter-layer Knowledge Distillation for Neural Machine Translation)—0
Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation—0
POS-Constrained Parallel Decoding for Non-autoregressive GenerationCode0
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation—0
Pose-Guided Feature Learning with Knowledge Distillation for Occluded Person Re-Identification—0
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation SignalsCode0
QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning—0
Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation—0
In Defense of the Learning Without Forgetting for Task Incremental Learning—0
Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation—0
ROD: Reception-aware Online Distillation for Sparse GraphsCode0
Show:102550
← PrevPage 141 of 170Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified