Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation

Code

• github.com/pgsmall/conceptbank
  OfficialIn paper★ 31

Abstract

The recent introduction of SAM3 has revolutionized Open-Vocabulary Segmentation (OVS) through promptable concept segmentation, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift (data drift) or conditional label distributions evolve (concept drift) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present ConceptBank, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach (i) anchors target-domain evidence via class-wise visual prototypes, (ii) mines representative supports to suppress outliers under data drift, and (iii) fuses candidate concepts to rectify concept drift. We demonstrate that ConceptBank effectively adapts SAM3 to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.

Reproductions