SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2901–2950 of 4240 papers

TitleStatusHype
Meta Knowledge Distillation—0
Knowledge Distillation with Deep SupervisionCode0
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation—0
FAMIE: A Fast Active Learning Framework for Multilingual Information ExtractionCode1
No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices—0
ZeroGen: Efficient Zero-shot Learning via Dataset GenerationCode1
Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Search—0
AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation—0
Tiny Object Tracking: A Large-scale Dataset and A BaselineCode2
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning—0
Point-Level Region Contrast for Object Detection Pre-TrainingCode1
Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationCode1
Adaptive Mixing of Auxiliary Losses in Supervised LearningCode0
Locally Differentially Private Distributed Deep Learning via Knowledge DistillationCode0
Measuring and Reducing Model Update Regression in Structured Prediction for NLP—0
Cross domain knowledge compression in realtime optical flow prediction on ultrasound sequences—0
Bootstrapped Representation Learning for Skeleton-Based Action Recognition—0
Iterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition—0
Local Feature Matching with Transformers for low-end devicesCode1
Deep-Disaster: Unsupervised Disaster Detection and Localization Using Visual DataCode0
Improving Robustness by Enhancing Weak SubnetsCode0
Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning—0
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models—0
Global-Reasoned Multi-Task Learning Model for Surgical Scene UnderstandingCode1
Dynamic Rectification Knowledge DistillationCode0
Anomaly Detection via Reverse Distillation from One-Class EmbeddingCode2
Adaptive Instance Distillation for Object Detection in Autonomous Driving—0
TrustAL: Trustworthy Active Learning using Knowledge Distillation—0
One Student Knows All Experts Know: From Sparse to Dense—0
Attentive Task Interaction Network for Multi-Task LearningCode0
Jointly Learning Knowledge Embedding and Neighborhood Consensus with Relational Knowledge Distillation for Entity Alignment—0
Federated Unlearning with Knowledge Distillation—0
AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models—0
Image-to-Video Re-Identification via Mutual Discriminative Knowledge Transfer—0
Can Model Compression Improve NLP Fairness—0
UKD: Debiasing Conversion Rate Estimation via Uncertainty-regularized Knowledge Distillation—0
Improving Neural Machine Translation by Denoising Training—0
Continual Coarse-to-Fine Domain Adaptation in Semantic SegmentationCode0
It's All in the Head: Representation Knowledge Distillation through Classifier SharingCode1
Cross-modal Contrastive Distillation for Instructional Activity Anticipation—0
Knowledge Distillation as Self-Supervised Learning—0
Tree Knowledge Distillation for Compressing Transformer-Based Language Models—0
Learning Cross-Lingual IR from an English Retriever—0
Nearest Neighbor Knowledge Distillation for Neural Machine Translation—0
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing—0
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation—0
Re2G: Retrieve, Rerank, Generate—0
CL-ReKD: Cross-lingual Knowledge Distillation for Multilingual Retrieval Question Answering—0
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation—0
SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge DistillationCode1
Show:102550
← PrevPage 59 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified