SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 451–500 of 4240 papers

TitleStatusHype
Graph-less Collaborative FilteringCode1
Data-Free Knowledge Distillation via Feature Exchange and Activation Region ConstraintCode1
Graph Relation Distillation for Efficient Biomedical Instance SegmentationCode1
Data-Free Network Quantization With Adversarial Knowledge DistillationCode1
HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast ImagesCode1
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation MethodCode1
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic HashingCode1
AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian DetectionCode1
BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge DistillationCode1
Black-Box Attacks on Sequential Recommenders via Data-Free Model ExtractionCode1
Adaptive Multi-Teacher Knowledge Distillation with Meta-LearningCode1
Black-box Few-shot Knowledge DistillationCode1
Adaptive Multi-Teacher Multi-level Knowledge DistillationCode1
HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language ModelsCode1
Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' OutputsCode1
Understanding the Role of the Projector in Knowledge DistillationCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Data-Free Knowledge Distillation for Heterogeneous Federated LearningCode1
Human-Inspired Multi-Agent Navigation using Knowledge DistillationCode1
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature ConsolidationCode1
I^3 Retriever: Incorporating Implicit Interaction in Pre-trained Language Models for Passage RetrievalCode1
iDAT: inverse Distillation Adapter-TuningCode1
Implicit Chain of Thought Reasoning via Knowledge DistillationCode1
Improved Techniques for Training Adaptive Deep NetworksCode1
Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient DetectorsCode1
Boosting Light-Weight Depth Estimation Via Knowledge DistillationCode1
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge DistillationCode1
DA-Mamba: Domain Adaptive Hybrid Mamba-Transformer Based One-Stage Object DetectionCode1
Data Diversification: A Simple Strategy For Neural Machine TranslationCode1
3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous DrivingCode1
Improving Simultaneous Machine Translation with Monolingual DataCode1
Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport DistillationCode1
Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain TransferCode1
Advantage-Guided Distillation for Preference Alignment in Small Language ModelsCode1
Information Theoretic Representation DistillationCode1
Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingsCode1
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual LearningCode1
Instance Relation Graph Guided Source-Free Domain Adaptive Object DetectionCode1
Instruction Multi-Constraint Molecular Generation Using a Teacher-Student Large Language ModelCode1
BPKD: Boundary Privileged Knowledge Distillation For Semantic SegmentationCode1
Breaking Modality Gap in RGBT Tracking: Coupled Knowledge DistillationCode1
Discriminative and Consistent Representation DistillationCode1
Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose EstimationCode1
Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object DetectionCode1
CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as TeachersCode1
APSNet: Attention Based Point Cloud SamplingCode1
Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object DetectionCode1
KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge DistillationCode1
DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable LearnersCode1
Show:102550
← PrevPage 10 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified