Heartcare Suite: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding

Code

• github.com/dcdmllm/heartcare-suite
  OfficialIn paper★ 30

Abstract

Although electrocardiograms (ECG) play a dominant role in cardiovascular diagnosis and treatment, their intrinsic data forms and representational patterns pose significant challenges for medical multimodal large language models (Med-MLLMs) in achieving cross-modal semantic alignment. To address this gap, we propose Heartcare Suite, a unified ECG suite designed for dual signal-image modeling and understanding. (i) Heartcare-400K: We build a finegrained ECG instruction dataset on top of our data pipeline engine--HeartAgent--by integrating 12,170 high quality clinical ECG reports from top hospitals with open-source data; (ii) Heartcare-Bench: a systematic benchmark assessing performance of models in multi-perspective ECG understanding and cross-modal generalization, providing guidance for optimizing ECG comprehension models; (iii) HeartcareGPT: built upon a structure-aware discrete tokenizer Beat, we propose the DSPA (Dual Stream Projection Alignment) paradigm--a dual encoder projection alignment mechanism enabling joint optimizing and modeling native ECG signal-image within a shared feature space. Heartcare achieves consistent improvements across diverse ECG understanding tasks, validating both the effectiveness of the unified modeling paradigm and the necessity of a high-quality data pipeline, and establishing a methodological foundation for extending Med-MLLMs toward physiological signal domains. Our project is available at https://github.com/DCDmllm/Heartcare-Suite .

Reproductions