Predicting Cognitive Decline: A Multimodal AI Approach to Dementia Screening from Speech

Abstract

Recent progress has been made in detecting early stage dementia entirely through recordings of patient speech. Multimodal speech analysis methods were applied to the PROCESS challenge, which requires participants to use audio recordings of clinical interviews to predict patients as healthy control, mild cognitive impairment (MCI), or dementia and regress the patient's Mini-Mental State Exam (MMSE) scores. The approach implemented in this work combines acoustic features (eGeMAPS and Prosody) with embeddings from Whisper and RoBERTa models, achieving competitive results in both regression (RMSE: 2.7666) and classification (Macro-F1 score: 0.5774) tasks. Additionally, a novel two-tiered classification setup is utilized to better differentiate between MCI and dementia. Our approach achieved strong results on the test set, ranking seventh on regression and eleventh on classification out of thirty-seven teams, exceeding the baseline results.

Tasks

Reproductions