SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1951–2000 of 4240 papers

TitleStatusHype
Analyzing Compression Techniques for Computer Vision—0
On enhancing the robustness of Vision Transformers: Defensive DiffusionCode0
Towards Understanding and Improving Knowledge Distillation for Neural Machine TranslationCode0
AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference—0
Black-box Source-free Domain Adaptation via Two-stage Knowledge Distillation—0
GSB: Group Superposition Binarization for Vision Transformer with Limited Training SamplesCode0
A Lightweight Domain Adversarial Neural Network Based on Knowledge Distillation for EEG-based Cross-subject Emotion Recognition—0
Knowledge distillation with Segment Anything (SAM) model for Planetary Geological Mapping—0
Improving Continual Relation Extraction by Distinguishing Analogous SemanticsCode1
Long-Tailed Question Answering in an Open World—0
Serial Contrastive Knowledge Distillation for Continual Few-shot Relation ExtractionCode1
A Survey on the Robustness of Computer Vision Models against Common CorruptionsCode0
Explainable Knowledge Distillation for On-device Chest X-Ray Classification—0
Multi-Teacher Knowledge Distillation For Text Image Machine TranslationCode0
SRIL: Selective Regularization for Class-Incremental Learning—0
FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise HeterogeneityCode1
SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language ModelsCode1
DynamicKD: An Effective Knowledge Distillation via Dynamic Entropy Correction-Based Distillation for Gap Optimizing—0
Distilling Script Knowledge from Large Language Models for Constrained Language PlanningCode1
Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation—0
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge—0
Web Content Filtering through knowledge distillation of Large Language Models—0
Structural and Statistical Texture Knowledge Distillation for Semantic Segmentation—0
Distilled Mid-Fusion Transformer Networks for Multi-Modal Human Activity Recognition—0
Smaller3d: Smaller Models for 3D Semantic Segmentation Using Minkowski Engine and Knowledge Distillation MethodsCode0
Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with UncertaintyCode1
SCOTT: Self-Consistent Chain-of-Thought DistillationCode1
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target TrainingCode0
DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge DistillationCode1
Structure Aware Incremental Learning with Personalized Imitation Weights for Recommender Systems—0
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models—0
Detect, Distill and Update: Detect, Distill and Update: Learned DB Systems Facing Out of Distribution DataCode0
Refined Response Distillation for Class-Incremental Player DetectionCode0
Scaffolding a Student to Instill KnowledgeCode0
Multi-to-Single Knowledge Distillation for Point Cloud Semantic SegmentationCode0
CORSD: Class-Oriented Relational Self Distillation—0
Ensemble Modeling with Contrastive Knowledge Distillation for Sequential RecommendationCode0
Learning Human-Human Interactions in Images from Weak Textual Supervision—0
A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question AnsweringCode1
Shape-Net: Room Layout Estimation from Panoramic Images Robust to Occlusion using Knowledge Distillation with 3D Shapes as Additional Inputs—0
Class Attention Transfer Based Knowledge DistillationCode1
Improving Knowledge Distillation via Transferring Learning AbilityCode0
A Forward and Backward Compatible Framework for Few-shot Class-incremental Pill RecognitionCode0
Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving—0
Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic SegmentationCode1
Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation—0
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual GraphsCode1
Word Sense Induction with Knowledge Distillation from BERT—0
Attention Weighted Local DescriptorsCode1
Knowledge Distillation Under Ideal Joint Classifier Assumption—0
Show:102550
← PrevPage 40 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified