Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generation

Code

• github.com/kimyong95/guide-stable-diffusion
  OfficialIn paperpytorch★ 6

Abstract

Guided diffusion-model generation is a promising direction for customizing the generation process of a pre-trained diffusion model to address specific downstream tasks. Existing guided diffusion models either rely on training the guidance model with pre-collected datasets or require the objective functions to be differentiable. However, for most real-world tasks, offline datasets are often unavailable, and their objective functions are often not differentiable, such as image generation with human preferences, molecular generation for drug discovery, and material design. Thus, we need an online algorithm capable of collecting data during runtime and supporting a black-box objective function. Moreover, the query efficiency of the algorithm is also critical because the objective evaluation of the query is often expensive in real-world scenarios. In this work, we propose a novel and simple algorithm, Fast Direct, for query-efficient online black-box target generation. Our Fast Direct builds a pseudo-target on the data manifold to update the noise sequence of the diffusion model with a universal direction, which is promising to perform query-efficient guided generation. Extensive experiments on twelve high-resolution ( 1024 1024) image target generation tasks and six 3D-molecule target generation tasks show 6 up to 10 query efficiency improvement and 11 up to 44 query efficiency improvement, respectively. Our implementation is publicly available at: https://github.com/kimyong95/guide-stable-diffusion/tree/fast-direct

Tasks

Reproductions