SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1501–1550 of 4240 papers

TitleStatusHype
HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View DistillationCode0
On the Transferability of Visual Features in Generalized Zero-Shot LearningCode0
Hybrid Attention Model Using Feature Decomposition and Knowledge Distillation for Glucose ForecastingCode0
Hybrid Data-Free Knowledge DistillationCode0
Applying Knowledge Distillation to Improve Weed Mapping With DronesCode0
Chemical transformer compression for accelerating both training and inference of molecular modelingCode0
Distribution Aligned Semantics Adaption for Lifelong Person Re-IdentificationCode0
Facilitating NSFW Text Detection in Open-Domain Dialogue Systems via Knowledge DistillationCode0
Facilitating Pornographic Text Detection for Open-Domain Dialogue Systems via Knowledge Distillation of Large Language ModelsCode0
Distributed Soft Actor-Critic with Multivariate Reward Representation and Knowledge DistillationCode0
TinyBERT: Distilling BERT for Natural Language UnderstandingCode0
HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge DistillationCode0
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic SegmentationCode0
Image Recognition with Online Lightweight Vision Transformer: A SurveyCode0
Invariant debiasing learning for recommendation via biased imputationCode0
How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face RecognitionCode0
HiTSR: A Hierarchical Transformer for Reference-based Super-ResolutionCode0
Holistic White-light Polyp Classification via Alignment-free Dense Distillation of Auxiliary Optical ChromoendoscopyCode0
Highlight Every Step: Knowledge Distillation via Collaborative TeachingCode0
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image ClassificationCode0
Distill n' Explain: explaining graph neural networks using simple surrogatesCode0
Guiding Frame-Level CTC Alignments Using Self-knowledge DistillationCode0
Distilling Virtual Examples for Long-tailed RecognitionCode0
FAKD: Feature Augmented Knowledge Distillation for Semantic SegmentationCode0
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series DataCode0
GSB: Group Superposition Binarization for Vision Transformer with Limited Training SamplesCode0
A Dual-Contrastive Framework for Low-Resource Cross-Lingual Named Entity RecognitionCode0
Distilling the Undistillable: Learning from a Nasty TeacherCode0
Group Multi-View Transformer for 3D Shape Analysis with Spatial EncodingCode0
GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric LearningCode0
Distilling the Knowledge of Romanian BERTs Using Multiple TeachersCode0
Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain ConversationCode0
CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature SharingCode0
An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity RecognitionCode0
Greedy-layer Pruning: Speeding up Transformer Models for Natural Language ProcessingCode0
Graph Knowledge Distillation to Mixture of ExpertsCode0
Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and FusionCode0
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning StrategyCode0
Distilling Stereo Networks for Performant and Efficient Leaner NetworksCode0
Graph-based Knowledge Distillation by Multi-head Attention NetworkCode0
Cooperative Classification and Rationalization for Graph GeneralizationCode0
Gradient Knowledge Distillation for Pre-trained Language ModelsCode0
Graph Entropy Minimization for Semi-supervised Node ClassificationCode0
Spending Your Winning Lottery Better After Drawing ItCode0
GOTHAM: Graph Class Incremental Learning Framework under Weak SupervisionCode0
Goldfish: An Efficient Federated Unlearning FrameworkCode0
Answering Diverse Questions via Text Attached with Key Audio-Visual CluesCode0
GNN's Uncertainty Quantification using Self-DistillationCode0
Distilling Object Detectors With Global KnowledgeCode0
Goal-Conditioned Q-Learning as Knowledge DistillationCode0
Show:102550
← PrevPage 31 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified