SEARCHNARRATIVE_REVIEWresearch methods, biomedical research

narrative_review·research methods, biomedical research, systematic review, clinical trial, epidemiology·PMC10288577

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Progress in biomedical engineering (Bristol, England) · 9 authors, 6 centres

AI SUMMARY

FIDELITY 100%

This summary was generated by AI from a single paper. It has not been reviewed by a clinician and is not clinical advice. Verify against the source before acting on it.

AGENT 01 · STRUCTURED SUMMARY

TRACE TO SOURCE

“

TL;DR

Fusion strategies are divided into feature-level and decision-level categories.

Full summary

314 CHARS

Fusion strategies are divided into feature-level and decision-level categories. Decision-level fusion integrates predictions from unimodal models via averaging, voting, or meta-classifiers. Notable limitations include challenges with incomplete or missing modality data, model generalizability, and explainability.

STUDY TYPE

narrative_review

SPECIALTY

research methods

SUMMARISED BY

AI pipeline

FIDELITY CHECK

100% · A

ORIGINAL PAPER

PMC10288577

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Progress in biomedical engineering (Bristol, England)

READ THE FULL PAPER

ABSTRACT

Abstract The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions. Keywords: multimodal learning, fusion method, medical data, diagnosis and prognosis

CITATION EXPORT

Can Cui, Haichun Yang, Yaohong Wang et al.. Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in biomedical engineering (Bristol, England). (2023). https://doi.org/10.1088/2516-1091/acc2fe

AUTHORS

9 · 6 CENTRES

Can Cui, Haichun Yang, Yaohong Wang, Shilin Zhao, Zuhayr Asad, Lori A Coburn, Keith T Wilson, Bennett A Landman, Yuankai Huo

1Department of Computer Science, Vanderbilt University, Nashville, TN 37235, United States of America
2Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Center, Nashville, TN 37215, United States of America
3Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37215, United States of America
4Division of Gastroenterology Hepatology, and Nutrition, Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States of America
5Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37235, United States of America
6Veterans Affairs Tennessee Valley Healthcare System, Nashville, TN 37212, United States of America