Towards AI-Assisted Protocol Analysis in Design Research: Automating Question Labelling with GPT-4 According to Eris’ (2004) Taxonomy

Code

• github.com/ahmedshahriar/AI-Assisted-Protocol-Analysis-in-Design-Research
  none★ 0

Abstract

This study explores the potential of large language models (LLM)-based tools, specifically GPT-4 – a state-of-the-art language processing model - to assist in the analysis of verbal protocols of design. We focus on Eris’ taxonomy, a well-established framework that classifies questions asked by participants in a design-focused task according to three broad categories: low-level, deep reasoning, and generative design questions. Using a large dataset of pre-classified questions from design review meetings, a series of experiments test GPT-4’s capability in the categorization task and evaluate how different factors influence its precision. Results indicate that GPT-4 matches performance by human coders – a promising result for design researchers who can benefit from this tool with little prior natural language processing expertise. Overall, findings offer insights into the strengths and limitations of LLMs in this context and suggest directions for future research into the use of LLM-based tools in qualitative analyses of design activity.

Reproductions