Supervised Multimodal Bitransformers for Classifying Images and Text

Code

• github.com/facebookresearch/mmbt
  OfficialIn paperpytorch★ 0
• github.com/huggingface/transformers
  pytorch★ 158,292
• github.com/ThilinaRajapakse/simpletransformers
  pytorch★ 4,235
• github.com/IsaacRodgz/multimodal-transformers-movies
  pytorch★ 11
• github.com/adriangrepo/mmbt_lightning
  pytorch★ 0
• github.com/IsaacRodgz/mmbt_experiments
  pytorch★ 0

Abstract

Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.

Tasks

Benchmark Results

Reproductions