PCNN: Probable-Class Nearest-Neighbor Explanations Improve Fine-Grained Image Classification Accuracy for AIs and Humans
Giang, Nguyen, Valerie Chen, Mohammad Reza Taesiri, Anh Totti Nguyen
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/giangnguyen2412/PCNN-src-code-TMRL2024Officialpytorch★ 6
- github.com/anguyen8/nearest-neighbor-XAIOfficialIn paperpytorch★ 4
Abstract
Nearest neighbors (NN) are traditionally used to compute final decisions, e.g., in Support Vector Machines or k-NN classifiers, and to provide users with explanations for the model's decision. In this paper, we show a novel utility of nearest neighbors: To improve predictions of a frozen, pretrained image classifier C. We leverage an image comparator S that (1) compares the input image with NN images from the top-K most probable classes given by C; and (2) uses scores from S to weight the confidence scores of C to refine predictions. Our method consistently improves fine-grained image classification accuracy on CUB-200, Cars-196, and Dogs-120. Also, a human study finds that showing users our probable-class nearest neighbors (PCNN) reduces over-reliance on AI, thus improving their decision accuracy over prior work which only shows only the most-probable (top-1) class examples.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| CUB-200-2011 | ResNet-50 | Accuracy | 88.59 | — | Unverified |
| Stanford Cars | ResNet-50 | Accuracy | 91.06 | — | Unverified |
| Stanford Dogs | ResNet-50 | Accuracy | 86.31 | — | Unverified |