SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 451–500 of 4240 papers

TitleStatusHype
Improved Techniques for Training Adaptive Deep NetworksCode1
ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian SplattingCode1
Improving Continual Relation Extraction by Distinguishing Analogous SemanticsCode1
Learning Efficient Vision Transformers via Fine-Grained Manifold DistillationCode1
Distilled Split Deep Neural Networks for Edge-Assisted Real-Time SystemsCode1
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
Enhancing Low-Density EEG-Based Brain-Computer Interfaces with Similarity-Keeping Knowledge DistillationCode1
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic HashingCode1
AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian DetectionCode1
BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge DistillationCode1
Black-Box Attacks on Sequential Recommenders via Data-Free Model ExtractionCode1
Adaptive Multi-Teacher Knowledge Distillation with Meta-LearningCode1
Model LEGO: Creating Models Like Disassembling and Assembling Building BlocksCode1
Coaching a Teachable StudentCode1
Improving Simultaneous Machine Translation with Monolingual DataCode1
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation MethodCode1
Understanding the Role of the Projector in Knowledge DistillationCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Blockwisely Supervised Neural Architecture Search with Knowledge DistillationCode1
Distilling Holistic Knowledge with Graph Neural NetworksCode1
Distilling Knowledge from Refinement in Multiple Instance Detection NetworksCode1
Distilling Large Vision-Language Model with Out-of-Distribution GeneralizabilityCode1
Distilling Knowledge from Reader to Retriever for Question AnsweringCode1
Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech RecognitionCode1
Distilling Knowledge from Self-Supervised Teacher by Embedding Graph AlignmentCode1
Distilling Knowledge via Knowledge ReviewCode1
Boosting Light-Weight Depth Estimation Via Knowledge DistillationCode1
Distilling Knowledge via Intermediate ClassifiersCode1
CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual DistillationCode1
Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion ModelsCode1
Distilling Meta Knowledge on Heterogeneous Graph for Illicit Drug Trafficker Detection on Social MediaCode1
Intra-Document Cascading: Learning to Select Passages for Neural Document RankingCode1
Distilling Object Detectors via Decoupled FeaturesCode1
Distilling Out-of-Distribution Robustness from Vision-Language Foundation ModelsCode1
Distilling Object Detectors with Feature RichnessCode1
itKD: Interchange Transfer-based Knowledge Distillation for 3D Object DetectionCode1
Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingsCode1
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual LearningCode1
3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous DrivingCode1
JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis ModelsCode1
BPKD: Boundary Privileged Knowledge Distillation For Semantic SegmentationCode1
Breaking Modality Gap in RGBT Tracking: Coupled Knowledge DistillationCode1
KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp SegmentationCode1
KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and QuantizationCode1
Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object DetectionCode1
Distilling Visual Priors from Self-Supervised LearningCode1
Advantage-Guided Distillation for Preference Alignment in Small Language ModelsCode1
Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object DetectionCode1
Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine TranslationCode1
APSNet: Attention Based Point Cloud SamplingCode1
Show:102550
← PrevPage 10 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified