SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2751–2800 of 4240 papers

TitleStatusHype
A Survey on Model Compression for Large Language Models—0
A Survey on Recent Teacher-student Learning Studies—0
A Survey on Symbolic Knowledge Distillation of Large Language Models—0
A Survey on Transformer Compression—0
Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence—0
ADPS: Asymmetric Distillation Post-Segmentation for Image Anomaly Detection—0
Asymmetric Image Retrieval with Cross Model Compatible Ensembles—0
Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again—0
Asynchronous Convergence in Multi-Task Learning via Knowledge Distillation from Converged Tasks—0
Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation—0
A Technical Study into Small Reasoning Language Models—0
A Theoretical Analysis of Soft-Label vs Hard-Label Training in Neural Networks—0
A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition—0
Attention-Guided Answer Distillation for Machine Reading Comprehension—0
Attention-guided Feature Distillation for Semantic Segmentation—0
Attention is all you need for boosting graph convolutional neural network—0
AttentionLite: Towards Efficient Self-Attention Models for Vision—0
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models—0
Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model—0
Audio Representation Learning by Distilling Video as Privileged Information—0
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation—0
Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression—0
A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation—0
A Unified Framework for Continual Learning and Unlearning—0
A Unified Knowledge-Distillation and Semi-Supervised Learning Framework to Improve Industrial Ads Delivery Systems—0
A Unified Knowledge Distillation Framework for Deep Directed Graphical Models—0
AutoADR: Automatic Model Design for Ad Relevance—0
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models—0
AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models—0
AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family—0
Automated Channel Pruning with Learned Importance—0
Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation—0
Automatic Block-wise Pruning with Auxiliary Gating Structures for Deep Convolutional Neural Networks—0
Automatic Mixed-Precision Quantization Search of BERT—0
AUTOSUMM: Automatic Model Creation for Text Summarization—0
A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models—0
Aware of the History: Trajectory Forecasting with the Local Behavior Data—0
AWF: Adaptive Weight Fusion for Enhanced Class Incremental Semantic Segmentation—0
BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models—0
Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation—0
Knowledge Distillation for Human Action Anticipation—0
Baidu Neural Machine Translation Systems for WMT19—0
Balance Divergence for Knowledge Distillation—0
Balanced softmax cross-entropy for incremental learning with and without memory—0
Balancing Cost and Benefit with Tied-Multi Transformers—0
A predictive machine learning force field framework for liquid electrolyte development—0
BanglaEmbed: Efficient Sentence Embedding Models for a Low-Resource Language Using Cross-Lingual Distillation Techniques—0
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization—0
Bayesian-Optimized One-Step Diffusion Model with Knowledge Distillation for Real-Time 3D Human Motion Prediction—0
BD-KD: Balancing the Divergences for Online Knowledge Distillation—0
Show:102550
← PrevPage 56 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified