DiffuCom: A novel diffusion model for comment generation

Abstract

Given that the diffusion model has good advantages in diversity, an increasing number of researchers have recently begun exploring natural language generation. However, despite significant progress in modeling based on text diffusion, there are still several challenges to be overcome in comment tasks. On the one hand, the source sentences in the comment task are long. If each timestep in the diffusion carries and computes the source sentence, it will significantly increase the computational overhead. On the other hand, the diffusion model is essentially an iterative non-autoregressive model, which lacks target decomposition in the decoding. This makes vanilla crossattention prone to overlook local perceptual content, leading to the problem of poor correlation. To solve the aforementioned problem, we propose a novel diffusion model for comment generation (named DiffuCom). Specifically, DiffuCom employs the transformer- based encoder-decoder structure, encoding only once during the reverse diffusion process, effectively reducing computational overhead. Additionally, we design a context - aware cross -attention module (CACAM) that integrates global and local cross-attention information through a gating mechanism, which more accurately captures relevant fragment information in the source sentence and improves the correlation. Second, selfconditioning technology was introduced in the training process, which can effectively improve the quality of the samples. Furthermore, to enhance the controllability of DiffuCom, we also propose a blended control. It combines class-conditional and classifier guidance by utilizing the difference between training and inference. Extensive experiments on four comment datasets demonstrate that the proposed DiffuCom model can generate comments that are both diverse and relevant.

Tasks

Reproductions