Predictive care, i.e. data driven prediction and risk stratification on an individual level, enables prevention, early intervention and tailored therapies. Predictive care is considered a cornerstone for future healthcare. Multimodal large language models (MLLM) with their few-shot learning capabilities and ability to perform various tasks have high potential even for prediction tasks but remain black boxes.

?

The project COGITAMUS-Health focuses on basic considerations to use MLLM for predictions and use their ability for self-reasoning. We aim to create a systematic review on methods to elicit reasoning from MLLM on multimodal medical data and implement as well as evaluate a method for deterioration prediction from publicly available multimodal data (e.g. biosignals, electronical health records, text notes from MIMIC-IV) as an exemplary use-case for our MLLM-based prediction and reasoning.

?

?

The project is a collaboration between the Chair of Diagnostic Sensing at the University of Augsburg and the Laboratory of Diagnostic and Interventional Adaptive Imaging (IADI) (Prof. Julien Oster) at INSERM in France. The?Professorship of Clinical and Translational Research in Cardiology ( Prof. Dario Bongiovanni) is closely involved in the project.