SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1351–1400 of 4240 papers

TitleStatusHype
Continual Learning for Neural Machine Translation—0
AIDE: Agentically Improve Visual Language Model with Domain Experts—0
A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation—0
Continual Learning for Fake Audio Detection—0
Continual Learning for Class- and Domain-Incremental Semantic Segmentation—0
Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression—0
Continual Face Forgery Detection via Historical Distribution Preserving—0
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation—0
Continual Distillation Learning: Knowledge Distillation in Prompt-based Continual Learning—0
Continual Detection Transformer for Incremental Object Detection—0
AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation—0
Adam: Dense Retrieval Distillation with Adaptive Dark Examples—0
FedD2S: Personalized Data-Free Federated Knowledge Distillation—0
FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks—0
Feature-map-level Online Adversarial Knowledge Distillation—0
AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting—0
Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classification—0
Audio Representation Learning by Distilling Video as Privileged Information—0
Contextual Knowledge Distillation for Transformer Compression—0
A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation—0
Contextualized Attention-based Knowledge Transfer for Spoken Conversational Question Answering—0
Contextual Distillation Model for Diversified Recommendation—0
Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model—0
Feature Kernel Distillation—0
Feature Structure Distillation for BERT Transferring—0
Contextual Affinity Distillation for Image Anomaly Detection—0
Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning—0
Feature Fusion and Knowledge-Distilled Multi-Modal Multi-Target Detection—0
Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation—0
A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning—0
Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations—0
Feature Interaction Fusion Self-Distillation Network For CTR Prediction—0
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models—0
Conformer with dual-mode chunked attention for joint online and offline ASR—0
Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation—0
Configurable Holography: Towards Display and Scene Adaptation—0
Confidence Preservation Property in Knowledge Distillation Abstractions—0
AttentionLite: Towards Efficient Self-Attention Models for Vision—0
Ada-DQA: Adaptive Diverse Quality-aware Feature Acquisition for Video Quality Assessment—0
ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation—0
Confidence Conditioned Knowledge Distillation—0
Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation—0
Attention is all you need for boosting graph convolutional neural network—0
Confidence Attention and Generalization Enhanced Distillation for Continuous Video Domain Adaptation—0
Attention-guided Feature Distillation for Semantic Segmentation—0
AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes—0
Conditional Generative Data-free Knowledge Distillation—0
Attention-Guided Answer Distillation for Machine Reading Comprehension—0
Conditional Autoregressors are Interpretable Classifiers—0
A Generative Framework for Personalized Learning and Estimation: Theory, Algorithms, and Privacy—0
Show:102550
← PrevPage 28 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified