BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Code

• github.com/salesforce/lavis
  OfficialIn paperpytorch★ 11,193
• github.com/huggingface/transformers
  pytorch★ 158,292
• github.com/thudm/visualglm-6b
  pytorch★ 4,166
• github.com/baaivision/eva
  pytorch★ 2,655
• github.com/junshutang/Make-It-3D
  pytorch★ 1,886
• github.com/facebookresearch/multimodal
  pytorch★ 1,706
• github.com/unispac/visual-adversarial-examples-jailbreak-large-language-models
  pytorch★ 269
• github.com/yukw777/videoblip
  pytorch★ 124
• github.com/alibaba/graphtranslator
  pytorch★ 118
• github.com/kdr/videorag-mrr2024
  none★ 32

Abstract

The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

Tasks

Benchmark Results

Reproductions