SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 101–150 of 4240 papers

TitleStatusHype
MI-GAN: A Simple Baseline for Image Inpainting on Mobile DevicesCode2
Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge DistillationCode2
SSDA-YOLO: Semi-supervised Domain Adaptive YOLO for Cross-Domain Object DetectionCode2
Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier TransformCode2
Let Images Give You More:Point Cloud Cross-Modal Training for Shape AnalysisCode2
On-Device Domain GeneralizationCode2
2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point CloudsCode2
MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized HealthcareCode2
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersCode2
Masked Generative DistillationCode2
Cross-Image Relational Knowledge Distillation for Semantic SegmentationCode2
Localization Distillation for Object DetectionCode2
Solving ImageNet: a Unified Scheme for Training any Backbone to Top ResultsCode2
Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise DistillationCode2
Decoupled Knowledge DistillationCode2
Tiny Object Tracking: A Large-scale Dataset and A BaselineCode2
Anomaly Detection via Reverse Distillation from One-Class EmbeddingCode2
MobileFaceSwap: A Lightweight Framework for Video Face SwappingCode2
LibFewShot: A Comprehensive Library for Few-shot LearningCode2
Semi-Supervised Domain Generalizable Person Re-IdentificationCode2
Learning Student Networks in the WildCode2
Knowledge distillation: A good teacher is patient and consistentCode2
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New OutlooksCode2
TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language ProcessingCode2
Scalable Zero-shot Entity Linking with Dense Entity RetrievalCode2
Positive-Unlabeled Compression on the CloudCode2
Well-Read Students Learn Better: On the Importance of Pre-training Compact ModelsCode2
Data-Free Knowledge Distillation for Deep Neural NetworksCode2
Focal Loss for Dense Object DetectionCode2
SeqPE: Transformer with Sequential Position EncodingCode1
SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM ReasoningCode1
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental LearningCode1
UWSAM: Segment Anything Model Guided Underwater Instance Segmentation and A Large-scale Benchmark DatasetCode1
Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge GraphsCode1
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation TrainerCode1
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank CloneCode1
Always Clear Depth: Robust Monocular Depth Estimation under Adverse WeatherCode1
Foundation Models Knowledge Distillation For Battery Capacity Degradation ForecastCode1
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-DivergenceCode1
Distribution-aware Forgetting Compensation for Exemplar-Free Lifelong Person Re-identificationCode1
Teach Me How to Denoise: A Universal Framework for Denoising Multi-modal Recommender Systems via Guided CalibrationCode1
A Dual-Space Framework for General Knowledge Distillation of Large Language ModelsCode1
Better Estimation of the KL Divergence Between Language ModelsCode1
Distillation and Refinement of Reasoning in Small Language Models for Document Re-rankingCode1
Multi-modal Knowledge Distillation-based Human Trajectory ForecastingCode1
High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous FlightCode1
Exploring Performance-Complexity Trade-Offs in Sound Event Detection ModelsCode1
Spatial Distillation based Distribution Alignment (SDDA) for Cross-Headset EEG ClassificationCode1
Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental LearningCode1
Advantage-Guided Distillation for Preference Alignment in Small Language ModelsCode1
Show:102550
← PrevPage 3 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified