Show Me a Story: Towards Coherent Neural Story Illustration

Code

• github.com/Hareesh-Ravi/Show-Me-A-Story
  Officialtf★ 0

Abstract

We propose an end-to-end network for the visual illustration of a sequence of sentences forming a story. At the core of our model is the ability to model the inter-related nature of the sentences within a story, as well as the ability to learn coherence to support reference resolution. The framework takes the form of an encoder-decoder architecture, where sentences are encoded using a hierarchical two-level sentence-story GRU, combined with an encoding of coherence, and sequentially decoded using predicted feature representation into a consistent illustrative image sequence. We optimize all parameters of our network in an end-to-end fashion with respect to order embedding loss, encoding entailment between images and sentences. Experiments on the VIST storytelling dataset citevist highlight the importance of our algorithmic choices and efficacy of our overall model.

Tasks

Reproductions