SOTAVerified

Knowledge Distillation

Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1851–1900 of 4240 papers

TitleStatusHype
Adaptive Multi-Teacher Knowledge Distillation with Meta-LearningCode1
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language ModelCode1
EaSyGuide : ESG Issue Identification Framework leveraging Abilities of Generative Large Language ModelsCode0
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation MethodCode1
Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition—0
RankFormer: Listwise Learning-to-Rank Using Listwide LabelsCode1
The economic trade-offs of large language models: A case study—0
BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping—0
Population-Based Evolutionary Gaming for Unsupervised Person Re-identification—0
Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level TasksCode1
Faithful Knowledge Distillation—0
Model-Based Reinforcement Learning with Multi-Task Offline PretrainingCode0
Orca: Progressive Learning from Complex Explanation Traces of GPT-4Code1
Zero shot framework for satellite image restoration—0
Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge GraphsCode0
I^3 Retriever: Incorporating Implicit Interaction in Pre-trained Language Models for Passage RetrievalCode1
Revisiting Data-Free Knowledge Distillation with Poisoned TeachersCode1
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference—0
Evolving Knowledge Mining for Class Incremental SegmentationCode0
Deep Classifier Mimicry without Data AccessCode0
Group channel pruning and spatial attention distilling for object detection—0
Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models—0
Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23—0
Improved Cross-Lingual Transfer Learning For Automatic Speech Translation—0
Teacher Agent: A Knowledge Distillation-Free Framework for Rehearsal-based Video Incremental LearningCode0
Accurate and Structured Pruning for Efficient Automatic Speech Recognition—0
Graph Entropy Minimization for Semi-supervised Node ClassificationCode0
PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) PlanningCode1
KEYword based Sampling (KEYS) for Large Language Models—0
Are Large Kernels Better Teachers than Transformers for ConvNets?Code2
Research on Multilingual News Clustering Based on Cross-Language Word Embeddings—0
Semi-supervised Pathological Image Segmentation via Cross Distillation of Multiple AttentionsCode1
A Recipe for Efficient SBIR Models: Combining Relative Triplet Loss with Batch Normalization and Knowledge Distillation—0
Bridging the Sim-to-Real Gap from the Information Bottleneck PerspectiveCode0
GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking—0
Learning to Learn from APIs: Black-Box Data-Free Meta-LearningCode1
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech ModelsCode1
ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval—0
Towards Better Entity Linking with Multi-View Enhanced DistillationCode1
FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image RecognitionCode1
One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker VerificationCode1
Knowledge Distillation Performs Partial Variance ReductionCode0
Vision Transformers for Small Histological Datasets Learned through Knowledge DistillationCode0
Improving Knowledge Distillation via Regularizing Feature Norm and DirectionCode1
ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression—0
A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models—0
Knowledge Diffusion for DistillationCode1
Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages—0
OVO: Open-Vocabulary OccupancyCode1
On the Impact of Knowledge Distillation for Model Interpretability—0
Show:102550
← PrevPage 38 of 85Next →
All datasetsImageNetCIFAR-100COCO (Common Objects in Context)COCO 2017 valPASCAL VOCKITTI

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1ScaleKD (T:BEiT-L S:ViT-B/14)Top-1 accuracy %86.43—Unverified
2ScaleKD (T:Swin-L S:ViT-B/16)Top-1 accuracy %85.53—Unverified
3ScaleKD (T:Swin-L S:ViT-S/16)Top-1 accuracy %83.93—Unverified
4ScaleKD (T:Swin-L S:Swin-T)Top-1 accuracy %83.8—Unverified
5KD++(T: regnety-16GF S:ViT-B)Top-1 accuracy %83.6—Unverified
6VkD (T:RegNety 160 S:DeiT-S)Top-1 accuracy %82.9—Unverified
7SpectralKD (T:Swin-S S:Swin-T)Top-1 accuracy %82.7—Unverified
8ScaleKD (T:Swin-L S:ResNet-50)Top-1 accuracy %82.55—Unverified
9DiffKD (T:Swin-L S: Swin-T)Top-1 accuracy %82.5—Unverified
10DIST (T: Swin-L S: Swin-T)Top-1 accuracy %82.3—Unverified
#ModelMetricClaimedVerifiedStatus
1SRD (T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)79.86—Unverified
2shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)78.76—Unverified
3MV-MR (T: CLIP/ViT-B-16 S: resnet50)Top-1 Accuracy (%)78.6—Unverified
4resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)78.28—Unverified
5resnet8x4 (T: resnet32x4 S: resnet8x4 [modified])Top-1 Accuracy (%)78.08—Unverified
6ReviewKD++(T:resnet-32x4, S:shufflenet-v2)Top-1 Accuracy (%)77.93—Unverified
7ReviewKD++(T:resnet-32x4, S:shufflenet-v1)Top-1 Accuracy (%)77.68—Unverified
8resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)77.5—Unverified
9resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.68—Unverified
10resnet8x4 (T: resnet32x4 S: resnet8x4)Top-1 Accuracy (%)76.31—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP77.16—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP73.73—Unverified
3ADLIK-Faster (T: Faster R-CNN vit-base S: Faster R-CNN deit-small) box AP47.6—Unverified
4ADLIK-Mask (T: Mask R-CNN vit-base S: Mask R-CNN deit-small)mask AP42.4—Unverified
#ModelMetricClaimedVerifiedStatus
1ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))AP@0.561.8—Unverified
2ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))AP@0.557.96—Unverified
3ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))AP@0.555.18—Unverified
#ModelMetricClaimedVerifiedStatus
1LSHFM (T: ResNet101 S: ResNet50)mAP93.17—Unverified
2LSHFM (T: ResNet101 S: MobileNetV2)mAP90.14—Unverified
#ModelMetricClaimedVerifiedStatus
1TIE-KD (T: Adabins S: MobileNetV2)RMSE2.43—Unverified