SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1401–1450 of 4240 papers

TitleStatusHype
Improving Question Answering Performance Using Knowledge Distillation and Active LearningCode0
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from EnsemblesCode0
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review MechanismCode0
Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object DetectionCode0
DSMix: Distortion-Induced Sensitivity Map Based Pre-training for No-Reference Image Quality AssessmentCode0
DSG-KD: Knowledge Distillation from Domain-Specific to General Language ModelsCode0
DS_FusionNet: Dynamic Dual-Stream Fusion with Bidirectional Knowledge Distillation for Plant Disease RecognitionCode0
Improving generalizability of distilled self-supervised speech processing models under distorted settingsCode0
Improving Knowledge Distillation via Transferring Learning AbilityCode0
DROP: Poison Dilution via Knowledge Distillation for Federated LearningCode0
Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic TranscriptsCode0
Improving Robustness by Enhancing Weak SubnetsCode0
Ensemble diverse hypotheses and knowledge distillation for unsupervised cross-subject adaptationCode0
Ensemble Knowledge Distillation for Learning Improved and Efficient NetworksCode0
Improving Neural Architecture Search Image Classifiers via Ensemble LearningCode0
Do You Remember . . . the Future? Weak-to-Strong generalization in 3D Object DetectionCode0
Improved Knowledge Distillation for Crowd Counting on IoT DeviceCode0
Improved Knowledge Distillation via Teacher AssistantCode0
IE-GAN: An Improved Evolutionary Generative Adversarial Network Using a New Fitness Function and a Generic Crossover OperatorCode0
Improving Adversarial Robust Fairness via Anti-Bias Soft Label DistillationCode0
Are All Linear Regions Created Equal?Code0
Image Recognition with Online Lightweight Vision Transformer: A SurveyCode0
Hybrid Data-Free Knowledge DistillationCode0
Hybrid Attention Model Using Feature Decomposition and Knowledge Distillation for Glucose ForecastingCode0
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic SegmentationCode0
HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View DistillationCode0
Class Incremental Fault Diagnosis under Limited Fault Data via Supervised Contrastive Knowledge DistillationCode0
Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation NetworksCode0
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model CompressionCode0
Attention to detail: inter-resolution knowledge distillationCode0
Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary DistillationCode0
How to Train the Teacher Model for Effective Knowledge DistillationCode0
HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge DistillationCode0
Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to EvolvabilityCode0
How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face RecognitionCode0
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning StrategyCode0
Domain Generalization for Crop Segmentation with Standardized Ensemble Knowledge DistillationCode0
HiTSR: A Hierarchical Transformer for Reference-based Super-ResolutionCode0
Highlight Every Step: Knowledge Distillation via Collaborative TeachingCode0
Domain Adaptable Fine-Tune Distillation Framework For Advancing Farm SurveillanceCode0
DOGe: Defensive Output Generation for LLM Protection Against Knowledge DistillationCode0
Does Training with Synthetic Data Truly Protect Privacy?Code0
Holistic White-light Polyp Classification via Alignment-free Dense Distillation of Auxiliary Optical ChromoendoscopyCode0
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image ClassificationCode0
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog SystemsCode0
CKD: Contrastive Knowledge Distillation from A Sample-wise PerspectiveCode0
DMSSN: Distilled Mixed Spectral-Spatial Network for Hyperspectral Salient Object DetectionCode0
Low-Cost Self-Ensembles Based on Multi-Branch Transformation and Grouped ConvolutionCode0
Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and FusionCode0
Group Multi-View Transformer for 3D Shape Analysis with Spatial EncodingCode0
Show:102550
← PrevPage 29 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified