AIDA - Prin [2022–2026]

AIDA: explAinable multImodal Deep learning for personAlized oncology

Grant number: 20228MZFAA-AIDA (CUP C53D23003620008)

In the fourth industrial revolution we are witnessing a fast and widespread adoption of artificial intelligence (AI) in our life, healthcare included. Advancements in deep learning (DL) should make significant contributions in this area supporting diagnosis, prognosis and treatment decisions. Most of the DL models consider only unimodal data, neglecting information available in other modalities of patient digital phenotypes, e.g. histopathology images and electronic health records (EHRs). As interpreting imaging findings is multimodal by its very nature, AI needs to be able to interpret such modalities together to progress towards higher informative clinical decision making.

AIDA tackles this challenge advancing multimodal DL (MDL), an area of great interest at its infancy. It studies how deep neural networks (DNNs) can learn shared representations between different modalities by investigating when to fuse the different modalities and how to embed in the training any process able to learn more powerful data representations. AIDA also searches for an optimal MDL fusion architecture, robust to missing modalities or missing data, and it studies multimodal regularization to improve stability, algorithmic speed-up and to reduce overfitting. It also considers approaches mitigating training from scratch, even when datasets of reduced size are available as it happens in healthcare.

A key impediment to the use of DL-based systems in practice is their black-box nature that does not permit to directly explain the decisions taken. Explainable AI (XAI) is now attempting to improve trust and transparency, but its investigation in healthcare is in an early stage. AIDA explores XAI methods for MDL using attention mechanisms, generating textual and fine-grained visual explanations, creating and testing human-understandable concepts, providing counterfactual examples and enabling the interaction between the algorithm and the final user. This can have a disruptive impact since model opacity makes it difficult for doctors and regulators to trust it.

Further to offer AI researchers a framework for MDL yielding a richer and trustworthy data representation producing much improved performance compared to use a single modality, AIDA tackles the association between radiomic, pathomic and EHRs in precision oncology to predict the patient outcomes in terms of progression free survival, overall survival, relapse time and response rate in non-small cell lung cancer, representing the 85% of all lung cancer cases. This represents one of the first attempts to integrate all such modalities as the literature still overlooks potential useful information they gather. AIDA performs a prospective study in two hospitals, experimentally evaluating the developed framework even in comparison with conventional clinical markers. This application in precision oncology strengthens the social and economic project impact, also aligned with the pillars of the Horizon Europe program.

Objectives

By interfacing several areas of AI, AIDA pursues the following general objectives: i) to investigate methods for MDL, studying when and how to learn shared representations, comparing with UDL; ii) to investigate methods to explain decisions taken by multimodal DNNs, exploiting attention mechanisms, textual and fine-grained visual explanations, counterfactual explanation and human-understandable concepts; iii) to deploy the breakthrough nature of MDL in personalized oncology, fusing radiomic, pathomic and EHRs data to predict treatment responses. 

These general objectives introduce also AIDA counterpart goals. They are: i) to perform a prospective study with patients affected by NSCLC, i.e. one of the major killer, since a publicly available repository with such multimodal data does not exist; ii) to experimentally validate the MDL methods developed on such prospective data and also on retrospective data already available in the PI’s unit, comparing also with conventional clinical markers.

Who we are

Visual abstract

List of Journal Articles

• Caruso, C. M., Soda, P., & Guarrasi, V. (2026). Not another imputation method: A transformer-based model for missing values in tabular datasets. AI Open.
• Paolo, D., Greco, C., Ippolito, E., Fiore, M., Ramella, S., Soda, P., … & Sicilia, R. (2026). Predicting lung cancer survival with attention-based CT slices combination. Health Information Science and Systems, 14(1), 20.
• Cortellini, A., Santo, V., Brunetti, L., Garbo, E., Pinato, D. J., La Cava, G., … & Aerts, J. G. (2025). Transformer-based AI approach to unravel long-term, time-dependent prognostic complexity in patients with advanced NSCLC and PD-L1 ≥ 50%: Insights from the pembrolizumab 5-year global registry. Journal for Immunotherapy of Cancer, 13(9), e012423.
• Caruso, C. M., Soda, P., & Guarrasi, V. (2025). MARIA: A multimodal transformer model for incomplete healthcare data. Computers in Biology and Medicine, 196, 110843.
• Paolo, D., Greco, C., Cortellini, A., Ramella, S., Soda, P., Bria, A., & Sicilia, R. (2025). Hierarchical embedding attention for overall survival prediction in lung cancer from unstructured EHRs. BMC Medical Informatics and Decision Making, 25(1), 169.
• Di Feola, F., Tronchin, L., Guarrasi, V., & Soda, P. (2025). Multi-scale texture loss for CT denoising with GANs. AI Open.
• Aksu, F., Gelardi, F., Chiti, A., & Soda, P. (2025). Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PET. Pattern Recognition Letters, 193, 86–93.
• Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025). A systematic review of intermediate fusion in multimodal deep learning for biomedical applications. Image and Vision Computing, 158, 105509.
• Caruso, C. M., Guarrasi, V., Ramella, S., & Soda, P. (2024). A deep learning approach for overall survival prediction in lung cancer with missing values. Computer Methods and Programs in Biomedicine, 254, 108308.

List of Peer-Reviewed Conference Proceedings

• Caruso, C. M., Bruni, R., & Guarrasi, V. (2025, September). Mask-aware transformers enable robust learning from incomplete volumetric medical imaging. In International Conference on Image Analysis and Processing (pp. 175–186). Cham: Springer Nature Switzerland.
• Sicilia, R., Aksu, F., Bria, A., Caragliano, A. N., Caruso, C. M., Cordelli, E., … & Soda, P. (2025, June). Next-Gen Health: From multimodal AI to foundation models. In CEUR Workshop Proceedings (Vol. 4121). CEUR-WS.
• Guarrasi, V., Di Feola, F., Iannello, G., Iele, I., Shen, L., Mantegna, M., … & Soda, P. (2025, June). Medicine without boundaries: Generative AI for translating medical data across modalities. In CEUR Workshop Proceedings (Vol. 4121). CEUR-WS.
• Ayllón, E. M., Mantegna, M., Shen, L., Soda, P., Guarrasi, V., & Tortora, M. (2025, June). Can foundation models really segment tumors? A benchmarking odyssey in lung CT imaging. In 2025 IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS) (pp. 375–380). IEEE.
• Aksu, F., Gelardi, F., Chiti, A., & Soda, P. (2024, December). Toward a multimodal deep learning approach for histological subtype classification in NSCLC. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (pp. 6327–6333). IEEE.
• Mantegna, M., Tronchin, L., Tortora, M., & Soda, P. (2024, December). Benchmarking GAN-based vs classical data augmentation on biomedical images. In International Conference on Pattern Recognition (pp. 92–104). Cham: Springer Nature Switzerland.
• Paolo, D., Bria, A., Greco, C., Russano, M., Ramella, S., Soda, P., & Sicilia, R. (2024, May). Exploring negated entities for named entity recognition in Italian lung cancer clinical reports. In Studies in Health Technology and Informatics (Vol. 314, pp. 98–102).
• Adornato, C., Assolito, C., Cordelli, E., Di Feola, F., Guarrasi, V., Iannello, G., … & Tronchin, L. (2024, May). Virtual scanner: Leveraging resilient generative AI for radiological imaging in the era of medical digital twins. In CEUR Workshop Proceedings (Vol. 3762, pp. 12–17). CEUR-WS.
• Aksu, F., Bria, A., Caragliano, A. N., Caruso, C. M., Chen, W., Cordelli, E., … & Zollo, L. (2024, May). Towards AI-driven next generation personalized healthcare and well-being. In CEUR Workshop Proceedings (Vol. 3762, pp. 360–365). CEUR-WS.
• Paolo, D., Bria, A., Greco, C., Russano, M., Ramella, S., Soda, P., & Sicilia, R. (2023, December). Named entity recognition in Italian lung cancer clinical reports using transformers. In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (pp. 4101–4107). IEEE.