Creating an AI Observer: Generative Semantic Workspaces

Abstract

An experienced human Observer reading a document -- such as a crime report -- creates a succinct plot-like ``Working Memory'' comprising different actors, their prototypical roles and states at any point, their evolution over time based on their interactions, and even a map of missing Semantic parts anticipating them in the future. An equivalent AI Observer currently does not exist. We introduce the [G]enerative [S]emantic [W]orkspace (GSW) -- comprising an ``Operator'' and a ``Reconciler'' -- that leverages advancements in LLMs to create a generative-style Semantic framework, as opposed to a traditionally predefined set of lexicon labels. Given a text segment C_n that describes an ongoing situation, the Operator instantiates actor-centric Semantic maps (termed ``Workspace instance'' W_n). The Reconciler resolves differences between W_n and a ``Working memory'' M_n^* to generate the updated M_n+1^*. GSW outperforms well-known baselines on several tasks ( 94\% vs. FST, GLEN, BertSRL - multi-sentence Semantics extraction, 15\% vs. NLI-BERT, 35\% vs. QA). By mirroring the real Observer, GSW provides the first step towards Spatial Computing assistants capable of understanding individual intentions and predicting future behavior.

Tasks

Reproductions