SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2401–2450 of 4240 papers

TitleStatusHype
Leveraging Expert Models for Training Deep Neural Networks in Scarce Data Domains: Application to Offline Handwritten Signature Verification—0
A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models—0
Three Factors to Improve Out-of-Distribution Detection—0
Spatio-Temporal Branching for Motion Prediction using Motion IncrementsCode0
Towards Better Query Classification with Multi-Expert Knowledge Condensation in JD Ads Search—0
Ada-DQA: Adaptive Diverse Quality-aware Feature Acquisition for Video Quality Assessment—0
Subspace Distillation for Continual LearningCode0
Sampling to Distill: Knowledge Transfer from Open-World Data—0
Federated Learning for Data and Model Heterogeneity in Medical Imaging—0
Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?Code0
UPFL: Unsupervised Personalized Federated Learning towards New ClientsCode0
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs—0
Mitigating Cross-client GANs-based Attack in Federated Learning—0
A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation—0
HeteFedRec: Federated Recommender Systems with Model Heterogeneity—0
Distribution Shift Matters for Knowledge Distillation with Webly Collected Images—0
Model Compression Methods for YOLOv5: A Review—0
Quantized Feature Distillation for Network Quantization—0
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns ClusteringCode0
LightPath: Lightweight and Scalable Path Representation LearningCode0
Teach model to answer questions after comprehending the document—0
Knowledge Distillation for Object Detection: from generic to remote sensing datasets—0
Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic TranscriptsCode0
Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain—0
Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages—0
MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning—0
A Survey of Techniques for Optimizing Transformer Inference—0
Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning—0
SoccerKDNet: A Knowledge Distillation Framework for Action Recognition in Soccer Videos—0
DreamTeacher: Pretraining Image Backbones with Deep Generative Models—0
Regression-Oriented Knowledge Distillation for Lightweight Ship Orientation Angle Prediction with Optical Remote Sensing ImagesCode0
Frameless Graph Knowledge DistillationCode0
A metric learning approach for endoscopic kidney stone identification—0
The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework—0
Customizing Synthetic Data for Data-Free Student LearningCode0
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series DataCode0
On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation—0
Contextual Affinity Distillation for Image Anomaly Detection—0
Distilling Missing Modality Knowledge from Ultrasound for Endometriosis Diagnosis with Magnetic Resonance Images—0
KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation—0
Review helps learn better: Temporal Supervised Knowledge Distillation—0
Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation—0
Long-Tailed Continual Learning For Visual Food Recognition—0
Streaming egocentric action anticipation: An evaluation scheme and approach—0
Understanding the Overfitting of the Episodic Meta-training—0
A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning—0
Exploring Dual Model Knowledge Distillation for Anomaly Detection—0
Shoggoth: Towards Efficient Edge-Cloud Collaborative Real-Time Video Inference via Adaptive Online Learning—0
Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation—0
Accelerating Molecular Graph Neural Networks via Knowledge Distillation—0
Show:102550
← PrevPage 49 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified