SOTAVerified

ShapeLLM: Universal 3D Object Understanding for Embodied Interaction

2024-02-27Code Available3· sign in to hype

Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, Kaisheng Ma

arXivPDF

Code Available — Be the first to reproduce this paper.

Reproduce

Code

Abstract

This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/

Tasks

3D geometry3D Object Captioning3D Point Cloud Classification3D Point Cloud Linear Classification3D Question Answering (3D-QA)Few-Shot 3D Point Cloud ClassificationGenerative 3D Object ClassificationInstruction FollowingLanguage ModelingLanguage ModellingLarge Language ModelMM-VetMultimodal Large Language ModelObjectVisual GroundingZero-shot 3D classificationZero-Shot Transfer 3D Point Cloud Classification

Benchmark Results

DatasetModelMetricClaimedVerifiedStatus
ObjaverseShapeLLM-7BGPT-446.92—Unverified
ObjaverseShapeLLM-13BGPT-448.94—Unverified

Reproductions