Transformer-Based Multimodal Fusion for Survival Prediction By Intergrating Whole Slide Images, Clinical, And Genomic Data

Survival prediction using whole slide images (WSIs) is a complex and difficult task, as handling gigapixel WSI directly is computationally impossible. In the past few years, people have worked out multiple instance learning (MIL) strategy to deal with WSIs, i.e., splitting WSI into many patches (instances) and aggregating features across patches. Moreover, to better predict survival outcome of patients, different modalities have been explored, among which gene features are used the most frequently. In this paper, we explore a graph-based strategy to handle WSIs and investigate a transformer-based strategy to combine different modalities for survival prediction. Moreover, clinical data was also adopted and different encoding manners of clinical information were explored. Experiments on two public TCGA datasets demonstrate the effectiveness of the proposed graph-transformer framework for survival prediction.