SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 2601–2650 of 4240 papers

TitleStatusHype
Scalable Syntax-Aware Language Models Using Knowledge Distillation—0
Scale-Equivalent Distillation for Semi-Supervised Object Detection—0
ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector—0
ScaleOT: Privacy-utility-scalable Offsite-tuning with Dynamic LayerReplace and Selective Rank Compression—0
Scaling Fair Learning to Hundreds of Intersectional Groups—0
Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis—0
Scaling Laws for Data-Efficient Visual Transfer Learning—0
Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective—0
SCARF: Scalable Continual Learning Framework for Memory-efficient Multiple Neural Radiance Fields—0
Scavenging Hyena: Distilling Transformers into Long Convolution Models—0
Scene-adaptive and Region-aware Multi-modal Prompt for Open Vocabulary Object Detection—0
Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search—0
Scene-aware Human Pose Generation using Transformer—0
Scene Graph Aided Radiology Report Generation—0
Scheduled Knowledge Acquisition on Lightweight Vector Symbolic Architectures for Brain-Computer Interfaces—0
Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA—0
SCLIFD:Supervised Contrastive Knowledge Distillation for Incremental Fault Diagnosis under Limited Fault Data—0
SDBERT: SparseDistilBERT, a faster and smaller BERT model—0
SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection—0
SDQ: Stochastic Differentiable Quantization with Mixed Precision—0
Search for Better Students to Learn Distilled Knowledge—0
Searching for COMETINHO: The Little Metric That Could—0
Search to Distill: Pearls are Everywhere but not the Eyes—0
SeCoKD: Aligning Large Language Models for In-Context Learning with Fewer Shots—0
Secost: Sequential co-supervision for large scale weakly labeled audio event detection—0
Secure Your Ride: Real-time Matching Success Rate Prediction for Passenger-Driver Pairs—0
SEDD-PCC: A Single Encoder-Dual Decoder Framework For End-To-End Learned Point Cloud Compression—0
Segment Any RGB-Thermal Model with Language-aided Distillation—0
SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models—0
Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models—0
Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning—0
SelectiveKD: A semi-supervised framework for cancer detection in DBT through Knowledge Distillation and Pseudo-labeling—0
Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation—0
Self-Cooperation Knowledge Distillation for Novel Class Discovery—0
Self-Distillation Amplifies Regularization in Hilbert Space—0
Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks—0
Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation—0
Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification—0
Self-Distilled Pruning Of Neural Networks—0
Self-Distilled Pruning of Neural Networks—0
Self-Evolution Knowledge Distillation for LLM-based Machine Translation—0
A New Training Framework for Deep Neural Network—0
SELF-KNOWLEDGE DISTILLATION ADVERSARIAL ATTACK—0
Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images—0
Self-Knowledge Distillation for Learning Ambiguity—0
Self-Knowledge Distillation for Surgical Phase Recognition—0
Self-Knowledge Distillation in Natural Language Processing—0
Self-Knowledge Distillation via Dropout—0
Self-Referenced Deep Learning—0
Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation—0
Show:102550
← PrevPage 53 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified