SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 451500 of 4240 papers

TitleStatusHype
Graph-less Collaborative FilteringCode1
Data-Free Class-Incremental Hand Gesture RecognitionCode1
Graph Relation Distillation for Efficient Biomedical Instance SegmentationCode1
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation MethodCode1
HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast ImagesCode1
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable LearnersCode1
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic HashingCode1
AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian DetectionCode1
BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge DistillationCode1
Black-Box Attacks on Sequential Recommenders via Data-Free Model ExtractionCode1
Adaptive Multi-Teacher Knowledge Distillation with Meta-LearningCode1
Black-box Few-shot Knowledge DistillationCode1
Adaptive Multi-Teacher Multi-level Knowledge DistillationCode1
HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language ModelsCode1
Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' OutputsCode1
Understanding the Role of the Projector in Knowledge DistillationCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Data-Free Knowledge Distillation for Heterogeneous Federated LearningCode1
Human-Inspired Multi-Agent Navigation using Knowledge DistillationCode1
I^3 Retriever: Incorporating Implicit Interaction in Pre-trained Language Models for Passage RetrievalCode1
IDa-Det: An Information Discrepancy-aware Distillation for 1-bit DetectorsCode1
Implicit Chain of Thought Reasoning via Knowledge DistillationCode1
Improve Cross-Architecture Generalization on Dataset DistillationCode1
Class-Balanced Distillation for Long-Tailed Visual RecognitionCode1
Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient DetectorsCode1
Boosting Light-Weight Depth Estimation Via Knowledge DistillationCode1
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge DistillationCode1
Dark Experience for General Continual Learning: a Strong, Simple BaselineCode1
3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous DrivingCode1
Improving Simultaneous Machine Translation with Monolingual DataCode1
DARTS: Double Attention Reference-based Transformer for Super-resolutionCode1
Advantage-Guided Distillation for Preference Alignment in Small Language ModelsCode1
Incremental Object Detection via Meta-LearningCode1
Information Theoretic Representation DistillationCode1
Informative knowledge distillation for image anomaly segmentationCode1
Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingsCode1
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual LearningCode1
Instruction Multi-Constraint Molecular Generation Using a Teacher-Student Large Language ModelCode1
Inter-Region Affinity Distillation for Road Marking SegmentationCode1
BPKD: Boundary Privileged Knowledge Distillation For Semantic SegmentationCode1
Breaking Modality Gap in RGBT Tracking: Coupled Knowledge DistillationCode1
Discriminative and Consistent Representation DistillationCode1
Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose EstimationCode1
Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object DetectionCode1
Jaccard Metric Losses: Optimizing the Jaccard Index with Soft LabelsCode1
APSNet: Attention Based Point Cloud SamplingCode1
Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object DetectionCode1
KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp SegmentationCode1
Curriculum Temperature for Knowledge DistillationCode1
Show:102550
← PrevPage 10 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified