SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Showing 301350 of 4240 papers

TitleStatusHype
CoNMix for Source-free Single and Multi-target Domain AdaptationCode1
ConStyle v2: A Strong Prompter for All-in-One Image RestorationCode1
Continual Learning for Image Segmentation with Dynamic QueryCode1
ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian SplattingCode1
Computation-Efficient Knowledge Distillation via Uncertainty-Aware MixupCode1
Comprehensive Knowledge Distillation with Causal InterventionCode1
Compressing Deep Graph Neural Networks via Adversarial Knowledge DistillationCode1
Confidence-Aware Multi-Teacher Knowledge DistillationCode1
A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language GuidanceCode1
A semi-supervised Teacher-Student framework for surgical tool detection and localizationCode1
SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object DetectorCode1
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge DistillationCode1
ConNER: Consistency Training for Cross-lingual Named Entity RecognitionCode1
Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy AnnotationsCode1
Designing Large Foundation Models for Efficient Training and Inference: A SurveyCode1
Content-Aware GAN CompressionCode1
Context-Aware Image Inpainting with Learned Semantic PriorsCode1
Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network StructureCode1
Adversarially Robust DistillationCode1
Contrastive Model Inversion for Data-Free Knowledge DistillationCode1
Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side DistillationCode1
Cross-category Video Highlight Detection via Set-based LearningCode1
Cross-Layer Distillation with Semantic CalibrationCode1
Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot ClassificationCode1
A Fast Knowledge Distillation Framework for Visual RecognitionCode1
Action knowledge for video captioning with graph neural networksCode1
Complementary Relation Contrastive DistillationCode1
Prototype-based Incremental Few-Shot Semantic SegmentationCode1
Conformer and Blind Noisy Students for Improved Image Quality AssessmentCode1
CTC-based Non-autoregressive Textless Speech-to-Speech TranslationCode1
Curriculum Temperature for Knowledge DistillationCode1
Dark Experience for General Continual Learning: a Strong, Simple BaselineCode1
DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech TranslationCode1
DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable LearnersCode1
A framework for benchmarking class-out-of-distribution detection and its application to ImageNetCode1
A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question AnsweringCode1
Continual Learning for LiDAR Semantic Segmentation: Class-Incremental and Coarse-to-Fine strategies on Sparse DataCode1
Data-Free Knowledge Distillation via Feature Exchange and Activation Region ConstraintCode1
Model LEGO: Creating Models Like Disassembling and Assembling Building BlocksCode1
Decomposed Knowledge Distillation for Class-Incremental Semantic SegmentationCode1
Decoupled Kullback-Leibler Divergence LossCode1
Decoupled Multimodal Distilling for Emotion RecognitionCode1
AgeFlow: Conditional Age Progression and Regression with Normalizing FlowsCode1
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation TrainerCode1
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-DivergenceCode1
Deep Structured Instance Graph for Distilling Object DetectorsCode1
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank CloneCode1
Deliberated Domain Bridging for Domain Adaptive Semantic SegmentationCode1
Dense Interspecies Face EmbeddingCode1
Coaching a Teachable StudentCode1
Show:102550
← PrevPage 7 of 85Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43Unverified