SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3851–3900 of 4240 papers

TitleStatusHype
Distilled Neural Networks for Efficient Learning to RankCode0
MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine TranslationCode0
MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph ClassificationCode0
Distilled Gradual Pruning with Pruned Fine-tuningCode0
ST-MFNet Mini: Knowledge Distillation-Driven Frame InterpolationCode0
Multi-aspect Knowledge Distillation with Large Language ModelCode0
Distilled GPT for Source Code SummarizationCode0
Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion SamplingCode0
Distill-DBDGAN: Knowledge Distillation and Adversarial Learning Framework for Defocus Blur DetectionCode0
Stolen Subwords: Importance of Vocabularies for Machine Translation Model StealingCode0
Multi-fidelity Neural Architecture Search with Knowledge DistillationCode0
StrassenNets: Deep Learning with a Multiplication BudgetCode0
DistillCSE: Distilled Contrastive Learning for Sentence EmbeddingsCode0
Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge DistillationCode0
Distillation Techniques for Pseudo-rehearsal Based Incremental LearningCode0
GOTHAM: Graph Class Incremental Learning Framework under Weak SupervisionCode0
Multi-granularity for knowledge distillationCode0
uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data RegimesCode0
Multi-Granularity Structural Knowledge Distillation for Language Model CompressionCode0
WARLearn: Weather-Adaptive Representation LearningCode0
Distillation Learning Guided by Image Reconstruction for One-Shot Medical Image SegmentationCode0
UFIN: Universal Feature Interaction Network for Multi-Domain Click-Through Rate PredictionCode0
Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to EvolvabilityCode0
Spending Your Winning Lottery Better After Drawing ItCode0
Goldfish: An Efficient Federated Unlearning FrameworkCode0
Goal-Conditioned Q-Learning as Knowledge DistillationCode0
Curriculum-scheduled Knowledge Distillation from Multiple Pre-trained Teachers for Multi-domain Sequential RecommendationCode0
GNN's Uncertainty Quantification using Self-DistillationCode0
GLiRA: Black-Box Membership Inference Attack via Knowledge DistillationCode0
GLANCE: Global to Local Architecture-Neutral Concept-based ExplanationsCode0
GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM DeploymentCode0
GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent InferenceCode0
Structural Knowledge Distillation: Tractably Distilling Information for Structured PredictorCode0
Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic SegmentationCode0
Multilingual Neural Machine Translation with Knowledge DistillationCode0
Multilingual Non-Autoregressive Machine Translation without Knowledge DistillationCode0
Automated Knowledge Distillation via Monte Carlo Tree SearchCode0
Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense PredictionCode0
Distillation Improves Visual Place Recognition for Low Quality ImagesCode0
Revisiting Distillation and Incremental Classifier LearningCode0
Generate, Annotate, and Learn: NLP with Synthetic TextCode0
Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge DistillationCode0
Multimodal Fusion SLAM with Fourier AttentionCode0
Multimodal Industrial Anomaly Detection by Crossmodal Reverse DistillationCode0
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting PerspectiveCode0
Generalizing Teacher Networks for Effective Knowledge Distillation Across Student ArchitecturesCode0
Generalized Knowledge Distillation via Relationship MatchingCode0
Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge DistillationCode0
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate LayersCode0
Revisiting Knowledge Distillation: An Inheritance and Exploration FrameworkCode0
Show:102550
← PrevPage 78 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified