SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1801–1850 of 4240 papers

TitleStatusHype
Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network—0
ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation—0
Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR—0
Handling Long-tailed Feature Distribution in AdderNets—0
Hands-on Guidance for Distilling Object Detectors—0
HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training—0
ADPS: Asymmetric Distillation Post-Segmentation for Image Anomaly Detection—0
HARD: Hard Augmentations for Robust Distillation—0
VizECGNet: Visual ECG Image Network for Cardiovascular Diseases Classification with Multi-Modal Training and Knowledge Distillation—0
Harmonizing knowledge Transfer in Neural Network with Unified Distillation—0
ELAICHI: Enhancing Low-resource TTS by Addressing Infrequent and Low-frequency Character Bigrams—0
ELAD: Explanation-Guided Large Language Models Active Distillation—0
EI-MTD:Moving Target Defense for Edge Intelligence against Adversarial Attacks—0
hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation—0
Headache to Overstock? Promoting Long-tail Items through Debiased Product Bundling—0
IOR: Inversed Objects Replay for Incremental Object Detection—0
Comparing Fisher Information Regularization with Distillation for DNN Quantization—0
Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networks—0
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation—0
HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression—0
HeteFedRec: Federated Recommender Systems with Model Heterogeneity—0
Heterogeneity-aware Personalized Federated Learning via Adaptive Dual-Agent Reinforcement Learning—0
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement—0
Heterogeneous Continual Learning—0
Incremental Learning for End-to-End Automatic Speech Recognition—0
Heterogeneous Federated Learning Using Knowledge Codistillation—0
ESGN: Efficient Stereo Geometry Network for Fast 3D Object Detection—0
Active Learning for Lane Detection: A Knowledge Distillation Approach—0
HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast—0
Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence—0
Improving Video Model Transfer With Dynamic Representation Learning—0
Hierarchical Knowledge Distillation on Text Graph for Data-limited Attribute Inference—0
Hierarchical Selective Classification—0
EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss—0
Compact Speaker Embedding: lrx-vector—0
Efficient Video Segmentation Models with Per-frame Inference—0
Efficient Verified Machine Unlearning For Distillation—0
Discovery of novel antimicrobial peptides with notable antibacterial potency by a LLM-based foundation model—0
Efficient Transformer Knowledge Distillation: A Performance Review—0
High Performance Natural Language Processing—0
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning—0
Hint-dynamic Knowledge Distillation—0
Compacting Deep Neural Networks for Internet of Things: Methods and Applications—0
Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation—0
Compact CNN Structure Learning by Knowledge Distillation—0
HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs—0
A Survey on Transformer Compression—0
Improving Text-based Early Prediction by Distillation from Privileged Time-Series Text—0
Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Study—0
Compact CNN Models for On-device Ocular-based User Recognition in Mobile Devices—0
Show:102550
← PrevPage 37 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified