SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 651–700 of 4240 papers

TitleStatusHype
Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational AgentsCode1
Focal and Global Knowledge Distillation for DetectorsCode1
Directed Acyclic Transformer for Non-Autoregressive Machine TranslationCode1
FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image RecognitionCode1
Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge GraphsCode1
Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?Code1
Prototype-based Incremental Few-Shot Semantic SegmentationCode1
General Cyclical Training of Neural NetworksCode1
Dense Interspecies Face EmbeddingCode1
COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using TransformersCode1
Generative Bias for Robust Visual Question AnsweringCode1
Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side DistillationCode1
Geometer: Graph Few-Shot Class-Incremental Learning via Prototype RepresentationCode1
Geometric Knowledge Distillation: Topology Compression for Graph Neural NetworksCode1
GlobalFlowNet: Video Stabilization using Deep Distilled Global Motion EstimatesCode1
Global Knowledge Calibration for Fast Open-Vocabulary SegmentationCode1
A Discrepancy Aware Framework for Robust Anomaly DetectionCode1
Good Teachers Explain: Explanation-Enhanced Knowledge DistillationCode1
Gradient-based Intra-attention Pruning on Pre-trained Language ModelsCode1
Graph-based Knowledge Distillation: A survey and experimental evaluationCode1
A framework for benchmarking class-out-of-distribution detection and its application to ImageNetCode1
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge DistillationCode1
A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question AnsweringCode1
Contrastive Model Inversion for Data-Free Knowledge DistillationCode1
Complementary Relation Contrastive DistillationCode1
Group Knowledge Transfer: Federated Learning of Large CNNs at the EdgeCode1
Deliberated Domain Bridging for Domain Adaptive Semantic SegmentationCode1
HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast ImagesCode1
Contrastive Distillation on Intermediate Representations for Language Model CompressionCode1
Heterogeneous Knowledge Distillation using Information Flow ModelingCode1
Hierarchical Self-supervised Augmented Knowledge DistillationCode1
Comprehensive Knowledge Distillation with Causal InterventionCode1
Densely Guided Knowledge Distillation using Multiple Teacher AssistantsCode1
Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' OutputsCode1
How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation ObjectivesCode1
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language UnderstandingCode1
Deep Structured Instance Graph for Distilling Object DetectorsCode1
AgeFlow: Conditional Age Progression and Regression with Normalizing FlowsCode1
I^3 Retriever: Incorporating Implicit Interaction in Pre-trained Language Models for Passage RetrievalCode1
IDa-Det: An Information Discrepancy-aware Distillation for 1-bit DetectorsCode1
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-DivergenceCode1
Improve Cross-Architecture Generalization on Dataset DistillationCode1
Improved Techniques for Training Adaptive Deep NetworksCode1
Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient DetectorsCode1
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank CloneCode1
Improving Knowledge Distillation via Regularizing Feature Norm and DirectionCode1
Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge DistillationCode1
Computation-Efficient Knowledge Distillation via Uncertainty-Aware MixupCode1
Defocus Blur Detection via Depth DistillationCode1
Camera clustering for scalable stream-based active distillationCode1
Show:102550
← PrevPage 14 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified