Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges
Hongru Wang, WenYu Huang, YuFei Wang, Yuanhao Xi, Jianqiao Lu, huan zhang, Nan Hu, Zeming Liu, Jeff Z. Pan, Kam-Fai Wong
Unverified — Be the first to reproduce this paper.
ReproduceAbstract
Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose DialogTool, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) tool creation; 2) tool utilization: tool awareness, tool selection, tool execution; and 3) role-consistent response: response generation and role play. Furthermore, we build VirtualMobile -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIsWe will use tools and APIs alternatively, there are no significant differences between them in this paper.. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons.