SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 501550 of 4240 papers

TitleStatusHype
Join the High Accuracy Club on ImageNet with A Binary Neural Network TicketCode1
Distilling Knowledge from Self-Supervised Teacher by Embedding Graph AlignmentCode1
DGEKT: A Dual Graph Ensemble Learning Method for Knowledge TracingCode1
Backdoor Cleansing with Unlabeled DataCode1
Multi-Level Knowledge Distillation for Out-of-Distribution Detection in TextCode1
Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge DistillationCode1
EEG aided boosting of single-lead ECG based sleep staging with Deep Knowledge DistillationCode1
BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object DetectionCode1
ConNER: Consistency Training for Cross-lingual Named Entity RecognitionCode1
FedCL: Federated Multi-Phase Curriculum Learning to Synchronously Correlate User HeterogeneityCode1
Cross-Modality Knowledge Distillation Network for Monocular 3D Object DetectionCode1
Fcaformer: Forward Cross Attention in Hybrid Vision TransformerCode1
MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge DistillationCode1
CoNMix for Source-free Single and Multi-target Domain AdaptationCode1
MPCFormer: fast, performant and private Transformer inference with MPCCode1
Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud AnalysisCode1
Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT ModelsCode1
Improved Feature Distillation via Projector EnsembleCode1
A Knowledge Distillation Framework For Enhancing Ear-EEG Based Sleep Staging With Scalp-EEG DataCode1
GlobalFlowNet: Video Stabilization using Deep Distilled Global Motion EstimatesCode1
Geometric Knowledge Distillation: Topology Compression for Graph Neural NetworksCode1
Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingsCode1
Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image RetrievalCode1
EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive PruningCode1
Learning Generalizable Models for Vehicle Routing Problems via Knowledge DistillationCode1
Efficient Knowledge Distillation from Model CheckpointsCode1
Decomposed Knowledge Distillation for Class-Incremental Semantic SegmentationCode1
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
APSNet: Attention Based Point Cloud SamplingCode1
ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot LearningCode1
Meta-Learning with Self-Improving Momentum TargetCode1
Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine TranslationCode1
Patch-based Knowledge Distillation for Lifelong Person Re-IdentificationCode1
Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-ExpertsCode1
IDa-Det: An Information Discrepancy-aware Distillation for 1-bit DetectorsCode1
Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image ClassificationCode1
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video RetrievalCode1
Effective Self-supervised Pre-training on Low-compute Networks without DistillationCode1
CLIP model is an Efficient Continual LearnerCode1
AlphaFold Distillation for Protein DesignCode1
Attention Distillation: self-supervised vision transformer students need more guidanceCode1
Multimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised LearningCode1
Hyper-Representations as Generative Models: Sampling Unseen Neural Network WeightsCode1
Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face RecognitionCode1
Efficient On-Device Session-Based RecommendationCode1
Rethinking Resolution in the Context of Efficient Video RecognitionCode1
Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak LabelsCode1
Deliberated Domain Bridging for Domain Adaptive Semantic SegmentationCode1
Switchable Online Knowledge DistillationCode1
Generative Adversarial Super-Resolution at the Edge with Knowledge DistillationCode1
Show:102550
← PrevPage 11 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified