SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 3601–3650 of 4240 papers

TitleStatusHype
On Knowledge Distillation for Direct Speech Translation—0
Progressive Network Grafting for Few-Shot Knowledge DistillationCode1
Distilling Knowledge from Reader to Retriever for Question AnsweringCode1
DE-RRD: A Knowledge Distillation Framework for Recommender SystemCode1
Model Compression Using Optimal Transport—0
Cross-Layer Distillation with Semantic CalibrationCode1
Multi-head Knowledge Distillation for Model Compression—0
Parallel Blockwise Knowledge Distillation for Deep Neural Network CompressionCode0
Reciprocal Supervised Learning Improves Neural Machine TranslationCode0
What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical PerspectiveCode1
Going Beyond Classification Accuracy Metrics in Model CompressionCode1
Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains—0
Query Distillation: BERT-based Distillation for Ensemble Ranking—0
Knowledge Base Embedding By Cooperative Knowledge DistillationCode1
Solvable Model for Inheriting the Regularization through Knowledge Distillation—0
Multi-level Knowledge Distillation via Knowledge Alignment and CorrelationCode1
Task-Oriented Feature DistillationCode1
Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Evolvability—0
Self-Supervised Generative Adversarial Compression—0
Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient SpaceCode1
Reverse-engineering recurrent neural network solutions to a hierarchical inference task for mice—0
Real-time Spatio-temporal Action Localization via Learning Motion Representation—0
Prototype-based Incremental Few-Shot Semantic SegmentationCode1
A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models—0
KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and QuantizationCode1
Channel-wise Knowledge Distillation for Dense PredictionCode1
Adaptive Multiplane Image Generation from a Single Internet Picture—0
torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation—0
Generative Adversarial Simulator—0
Multiresolution Knowledge Distillation for Anomaly DetectionCode1
Evolving Search Space for Neural Architecture SearchCode1
Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing SystemsCode1
MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing—0
KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge DistillationCode1
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation—0
Privileged Knowledge Distillation for Online Action Detection—0
A Knowledge Distillation Ensemble Framework for Predicting Short and Long-term Hospitalisation Outcomes from Electronic Health Records DataCode0
Deep Serial Number: Computational Watermarking for DNN Intellectual Property Protection—0
Generalized Continual Zero-Shot Learning—0
Digging Deeper into CRNN Model in Chinese Text Images Recognition—0
Anomaly Detection in Video via Self-Supervised and Multi-Task LearningCode1
Online Ensemble Model Compression using Knowledge DistillationCode0
EGAD: Evolving Graph Representation Learning with Self-Attention and Knowledge Distillation for Live Video Streaming EventsCode0
Real-Time Decentralized knowledge Transfer at the EdgeCode0
Distill2Vec: Dynamic Graph Representation Learning with Knowledge DistillationCode0
On Estimating the Training Cost of Conversational Recommendation Systems—0
Knowledge Distillation for Singing Voice DetectionCode0
Ensemble Knowledge Distillation for CTR Prediction—0
Human-Like Active Learning: Machines Simulating the Human Learning Process—0
Robustness and Diversity Seeking Data-Free Knowledge DistillationCode0
Show:102550
← PrevPage 73 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified