SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 801–850 of 4240 papers

TitleStatusHype
Distilling Linguistic Context for Language Model CompressionCode1
Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltyCode1
Backdoor Attacks on Self-Supervised LearningCode1
On-Device Next-Item Recommendation with Self-Supervised Knowledge DistillationCode1
Backdoor Cleansing with Unlabeled DataCode1
Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech RecognitionCode1
Distilling Image Classifiers in Object DetectorsCode1
Cross-Layer Distillation with Semantic CalibrationCode1
Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot ClassificationCode1
On information captured by neural networks: connections with memorization and generalizationCode1
Online Knowledge Distillation for Efficient Pose EstimationCode1
Online Knowledge Distillation via Collaborative LearningCode1
Online Prototype Learning for Online Continual LearningCode1
Online Speculative DecodingCode1
Distilling Knowledge from Reader to Retriever for Question AnsweringCode1
Fcaformer: Forward Cross Attention in Hybrid Vision TransformerCode1
Balanced Knowledge Distillation for Long-tailed LearningCode1
Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image RetrievalCode1
Cross-modality Data Augmentation for End-to-End Sign Language TranslationCode1
DARTS: Double Attention Reference-based Transformer for Super-resolutionCode1
Cross-Modality Knowledge Distillation Network for Monocular 3D Object DetectionCode1
Orca: Progressive Learning from Complex Explanation Traces of GPT-4Code1
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine TranslationCode1
Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response DistillationCode1
OVO: Open-Vocabulary OccupancyCode1
OVTrack: Open-Vocabulary Multiple Object TrackingCode1
Aligned Structured Sparsity Learning for Efficient Image Super-ResolutionCode1
PARADE: Passage Representation Aggregation for Document RerankingCode1
PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge DistillationCode1
CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image FusionCode1
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual InformationCode1
CTC-based Non-autoregressive Textless Speech-to-Speech TranslationCode1
DIOD: Self-Distillation Meets Object DiscoveryCode1
Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge DistillationCode1
Distilling Holistic Knowledge with Graph Neural NetworksCode1
Curriculum Learning for Dense Retrieval DistillationCode1
Curriculum Temperature for Knowledge DistillationCode1
BearingPGA-Net: A Lightweight and Deployable Bearing Fault Diagnosis Network via Decoupled Knowledge Distillation and FPGA AccelerationCode1
Digging into contrastive learning for robust depth estimation with diffusion modelsCode1
Extending global-local view alignment for self-supervised learning with remote sensing imageryCode1
PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) PlanningCode1
PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge DistillationCode1
Point, Segment and Count: A Generalized Framework for Object CountingCode1
Distilling Knowledge from Refinement in Multiple Instance Detection NetworksCode1
PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge DistillationCode1
Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuningCode1
Distilling Meta Knowledge on Heterogeneous Graph for Illicit Drug Trafficker Detection on Social MediaCode1
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video RetrievalCode1
A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect DetectionCode1
Distilling Cross-Task Knowledge via Relationship MatchingCode1
Show:102550
← PrevPage 17 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified