SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2951–3000 of 4240 papers

TitleStatusHype
Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment—0
Compressing Visual-linguistic Model via Knowledge Distillation—0
Compression of Acoustic Event Detection Models With Quantized Distillation—0
Compression of Deep Learning Models for Text: A Survey—0
Compression of end-to-end non-autoregressive image-to-speech system for low-resourced devices—0
ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval—0
ConceptDistil: Model-Agnostic Distillation of Concept Explanations—0
Condensed Sample-Guided Model Inversion for Knowledge Distillation—0
Conditional Autoregressors are Interpretable Classifiers—0
Conditional Generative Data-free Knowledge Distillation—0
Confidence Attention and Generalization Enhanced Distillation for Continuous Video Domain Adaptation—0
Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation—0
Confidence Conditioned Knowledge Distillation—0
Confidence Preservation Property in Knowledge Distillation Abstractions—0
Configurable Holography: Towards Display and Scene Adaptation—0
Conformer with dual-mode chunked attention for joint online and offline ASR—0
Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation—0
Contextual Affinity Distillation for Image Anomaly Detection—0
Contextual Distillation Model for Diversified Recommendation—0
Contextualized Attention-based Knowledge Transfer for Spoken Conversational Question Answering—0
Contextual Knowledge Distillation for Transformer Compression—0
Continual Detection Transformer for Incremental Object Detection—0
Continual Distillation Learning: Knowledge Distillation in Prompt-based Continual Learning—0
Continual Face Forgery Detection via Historical Distribution Preserving—0
Continual Learning for Class- and Domain-Incremental Semantic Segmentation—0
Continual Learning for Fake Audio Detection—0
Continual Learning for Neural Machine Translation—0
Unsupervised Continual Learning Via Pseudo Labels—0
Continual Learning with Diffusion-based Generative Replay for Industrial Streaming Data—0
Continual Learning with Dirichlet Generative-based Rehearsal—0
Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans—0
Continual Segment: Towards a Single, Unified and Non-forgetting Continual Segmentation Model of 143 Whole-body Organs in CT Scans—0
Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing—0
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization—0
Continuous Concepts Removal in Text-to-image Diffusion Models—0
Continuous sign language recognition based on cross-resolution knowledge distillation—0
Contrastive Continual Multi-view Clustering with Filtered Structural Fusion—0
Contrastive Learning-Based Spectral Knowledge Distillation for Multi-Modality and Missing Modality Scenarios in Semantic Segmentation—0
Contrastive Representation Distillation via Multi-Scale Feature Decoupling—0
Contrast R-CNN for Continual Learning in Object Detection—0
Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition—0
Controlling the Quality of Distillation in Response-Based Network Compression—0
Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies—0
Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition—0
Cooperative Denoising for Distantly Supervised Relation Extraction—0
Cooperative Learning for Cost-Adaptive Inference—0
Coordinating Cross-modal Distillation for Molecular Property Prediction—0
ChromaDistill: Colorizing Monochrome Radiance Fields with Knowledge Distillation—0
CoroNetGAN: Controlled Pruning of GANs via Hypernetworks—0
Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible—0
Show:102550
← PrevPage 60 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified