SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 851–900 of 4240 papers

TitleStatusHype
BERT-of-Theseus: Compressing BERT by Progressive Module ReplacingCode1
High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous FlightCode1
DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech TranslationCode1
DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable LearnersCode1
Data Diversification: A Simple Strategy For Neural Machine TranslationCode1
Better Estimation of the KL Divergence Between Language ModelsCode1
HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillationCode1
Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' OutputsCode1
Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport DistillationCode1
How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation ObjectivesCode1
Self-Knowledge Distillation with Progressive Refinement of TargetsCode1
Self-Mutual Distillation Learning for Continuous Sign Language RecognitionCode1
A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect DetectionCode1
Knowledge Distillation for Feature Extraction in Underwater VSLAMCode1
DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge DistillationCode1
Human-Inspired Multi-Agent Navigation using Knowledge DistillationCode1
BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object DetectionCode1
Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud AnalysisCode1
SimDistill: Simulated Multi-modal Distillation for BEV 3D Object DetectionCode1
Self-Supervised Vision Transformers Learn Visual Concepts in HistopathologyCode1
A Neural Span-Based Continual Named Entity Recognition ModelCode1
Distilling DETR with Visual-Linguistic Knowledge for Open-Vocabulary Object DetectionCode1
Hyper-Representations for Pre-Training and Transfer LearningCode1
Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action RecognitionCode1
Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training ModelCode1
Data-Free Knowledge Distillation via Feature Exchange and Activation Region ConstraintCode1
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT CompressionCode1
I^3 Retriever: Incorporating Implicit Interaction in Pre-trained Language Models for Passage RetrievalCode1
Knowledge Distillation for Multi-task LearningCode1
Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve BackbonesCode1
AlphaFold Distillation for Protein DesignCode1
Improved Feature Distillation via Projector EnsembleCode1
Decoupled Kullback-Leibler Divergence LossCode1
DA-Mamba: Domain Adaptive Hybrid Mamba-Transformer Based One-Stage Object DetectionCode1
Sequence-Level Knowledge DistillationCode1
Serial Contrastive Knowledge Distillation for Continual Few-shot Relation ExtractionCode1
Decoupled Multimodal Distilling for Emotion RecognitionCode1
Improving Continual Relation Extraction by Distinguishing Analogous SemanticsCode1
Improving Knowledge Distillation via Category StructureCode1
Simplified TinyBERT: Knowledge Distillation for Document RetrievalCode1
BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge DistillationCode1
Improving Event Detection via Open-domain Trigger KnowledgeCode1
AltDiffusion: A Multilingual Text-to-Image Diffusion ModelCode1
Three-Stream Temporal-Shift Attention Network Based on Self-Knowledge Distillation for Micro-Expression RecognitionCode1
Bidirectional Distillation for Top-K Recommender SystemCode1
Always Clear Depth: Robust Monocular Depth Estimation under Adverse WeatherCode1
Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image ClassificationCode1
Deep Structured Instance Graph for Distilling Object DetectorsCode1
Reducing the Teacher-Student Gap via Spherical Knowledge DisitllationCode1
Decomposed Knowledge Distillation for Class-Incremental Semantic SegmentationCode1
Show:102550
← PrevPage 18 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified