SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 251300 of 4240 papers

TitleStatusHype
AICSD: Adaptive Inter-Class Similarity Distillation for Semantic SegmentationCode1
Context-Aware Image Inpainting with Learned Semantic PriorsCode1
A Deep Knowledge Distillation framework for EEG assisted enhancement of single-lead ECG based sleep stagingCode1
Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network StructureCode1
CaKDP: Category-aware Knowledge Distillation and Pruning Framework for Lightweight 3D Object DetectionCode1
Distilling Knowledge from Graph Convolutional NetworksCode1
Distillation-Based Training for Multi-Exit ArchitecturesCode1
Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language ModelCode1
Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient SpaceCode1
DistilCSE: Effective Knowledge Distillation For Contrastive Sentence EmbeddingsCode1
Discriminator-Cooperated Feature Map Distillation for GAN CompressionCode1
AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric LearningCode1
Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object DetectionCode1
DisCo: Distilled Student Models Co-training for Semi-supervised Text MiningCode1
Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal PerspectiveCode1
Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage RetrievalCode1
BPKD: Boundary Privileged Knowledge Distillation For Semantic SegmentationCode1
Breaking Modality Gap in RGBT Tracking: Coupled Knowledge DistillationCode1
Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingsCode1
AdaDistill: Adaptive Knowledge Distillation for Deep Face RecognitionCode1
Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object DetectionCode1
Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation ModelsCode1
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterCode1
Digging into contrastive learning for robust depth estimation with diffusion modelsCode1
Boosting Light-Weight Depth Estimation Via Knowledge DistillationCode1
Extending global-local view alignment for self-supervised learning with remote sensing imageryCode1
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-DivergenceCode1
AgeFlow: Conditional Age Progression and Regression with Normalizing FlowsCode1
Boosting Multi-Label Image Classification with Complementary Parallel Self-DistillationCode1
DIOD: Self-Distillation Meets Object DiscoveryCode1
Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual LearningCode1
Block-Wisely Supervised Neural Architecture Search With Knowledge DistillationCode1
Blockwisely Supervised Neural Architecture Search with Knowledge DistillationCode1
A framework for benchmarking class-out-of-distribution detection and its application to ImageNetCode1
Dice Semimetric Losses: Optimizing the Dice Score with Soft LabelsCode1
DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic SegmentationCode1
Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge DistillationCode1
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic HashingCode1
BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge DistillationCode1
Hybrid Inverted Index Is a Robust Accelerator for Dense RetrievalCode1
Prototype-based Incremental Few-Shot Semantic SegmentationCode1
Black-Box Attacks on Sequential Recommenders via Data-Free Model ExtractionCode1
Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational AgentsCode1
A Fast Knowledge Distillation Framework for Visual RecognitionCode1
BiLD: Bi-directional Logits Difference Loss for Large Language Model DistillationCode1
Adversarially Robust DistillationCode1
Black-box Few-shot Knowledge DistillationCode1
DGEKT: A Dual Graph Ensemble Learning Method for Knowledge TracingCode1
DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue GenerationCode1
Directed Acyclic Transformer for Non-Autoregressive Machine TranslationCode1
Show:102550
← PrevPage 6 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified