Deep Visual-Linguistic Fusion Network Considering Cross-Modal Inconsistency for Rumor Detection

Code

• github.com/njustkmg/DVLFN
  Officialmindspore★ 0
• github.com/njustkmg/DVLFN_MindSpore
  mindspore★ 0
• github.com/lyqcom/Science-China_DVLFN-MindSpore
  mindspore★ 0

Abstract

With the development of internet, users can freely publish posts on various social media, which brings great convenience to keep abreast of the world. However, those posts usually carry a lot of rumors, which require plenty of manpower to monitor. Fortunately, thanks to the success of modern machine learning techniques, especially deep learning models, we tried to automatically detect the rumors as a classification problem. Early attempts always focus on building classifiers relying on image or text information, i.e., single modality in the posts. Thereafter, several multi-modal detection approaches employ early or late fusion operator for aggregating multiple source information. Nevertheless, they only take the advantage of multiple modal embeddings for fusion, ignore another important detection factor, i.e., the inter-modal inconsistency between modalities. To solve this problem, we develop the DVLFN, bluea novel Deep Visual-Linguistic Fusion Network considering cross-modal inconsistency, which detects rumors by comprehensively considering the modal aggregation and contrast information. Specifically, DVLFN firstly utilizes visual and textual deep encoders, i.e., BERT and Faster R-CNN, to extract global and regional embeddings for image and text modalities, then predicts the posts' authenticity from two aspects: 1) inter-modal inconsistency, which employs the wasserstein distance to better measure the similarity between regional embeddings of different modalities. 2) modal aggregation, which experimentally employs early fusion to aggregate two modal embeddings for prediction. Consequently, DVLFN can compose the final prediction based on the modal fusion and inconsistency measure. Experiments are conducted on three real-world multi-media rumor detection datasets collected from Reddit, GoodNews, and Weibo, the results validate the superior performances of the proposed DVLFN.

Reproductions