SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 201–250 of 4240 papers

TitleStatusHype
Designing Large Foundation Models for Efficient Training and Inference: A SurveyCode1
Content-Aware GAN CompressionCode1
Model LEGO: Creating Models Like Disassembling and Assembling Building BlocksCode1
Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network StructureCode1
Continual evaluation for lifelong learning: Identifying the stability gapCode1
Aligned Structured Sparsity Learning for Efficient Image Super-ResolutionCode1
AIM 2024 Challenge on UHD Blind Photo Quality AssessmentCode1
Contrastive Representation DistillationCode1
Cross-category Video Highlight Detection via Set-based LearningCode1
CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual DistillationCode1
Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language ModelCode1
Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model EnhancementCode1
CrossMatch: Enhance Semi-Supervised Medical Image Segmentation with Perturbation Strategies and Knowledge DistillationCode1
Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image RetrievalCode1
CTC-based Non-autoregressive Textless Speech-to-Speech TranslationCode1
Cumulative Spatial Knowledge Distillation for Vision TransformersCode1
Curriculum Temperature for Knowledge DistillationCode1
Dark Experience for General Continual Learning: a Strong, Simple BaselineCode1
AlphaFold Distillation for Protein DesignCode1
Collaborative Distillation for Ultra-Resolution Universal Style TransferCode1
Data Diversification: A Simple Strategy For Neural Machine TranslationCode1
AltDiffusion: A Multilingual Text-to-Image Diffusion ModelCode1
Always Clear Depth: Robust Monocular Depth Estimation under Adverse WeatherCode1
Data-Free Network Quantization With Adversarial Knowledge DistillationCode1
Decomposed Knowledge Distillation for Class-Incremental Semantic SegmentationCode1
Cloud Object Detector Adaptation by Integrating Different Source KnowledgeCode1
DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge DistillationCode1
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine TranslationCode1
AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian DetectionCode1
Adaptive Multi-Teacher Knowledge Distillation with Meta-LearningCode1
Adaptive Multi-Teacher Multi-level Knowledge DistillationCode1
Defocus Blur Detection via Depth DistillationCode1
Understanding the Role of the Projector in Knowledge DistillationCode1
Deliberated Domain Bridging for Domain Adaptive Semantic SegmentationCode1
APSNet: Attention Based Point Cloud SamplingCode1
Densely Guided Knowledge Distillation using Multiple Teacher AssistantsCode1
DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery CluesCode1
DGEKT: A Dual Graph Ensemble Learning Method for Knowledge TracingCode1
FocusNet: Classifying Better by Focusing on Confusing ClassesCode1
CLRKDNet: Speeding up Lane Detection with Knowledge DistillationCode1
Digging into contrastive learning for robust depth estimation with diffusion modelsCode1
Extending global-local view alignment for self-supervised learning with remote sensing imageryCode1
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual LearningCode1
AICSD: Adaptive Inter-Class Similarity Distillation for Semantic SegmentationCode1
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation MethodCode1
DisCo: Distilled Student Models Co-training for Semi-supervised Text MiningCode1
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterCode1
Distilling Knowledge from Graph Convolutional NetworksCode1
CLIP model is an Efficient Continual LearnerCode1
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental LearningCode1
Show:102550
← PrevPage 5 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified