Inter-Modal Conditional-Guided Fusion Network with Transformer for Grading Hepatocellular Carcinoma

Multimodal medical imaging plays an important role in the diagnosis and characterization of lesions. Transformer pays more attention to global relationship modeling in data, which has obtained promising performance in lesion characterization. However, there are still challenges in transformer-based multimodal feature fusion. First, simple concatenation of information from other modalities, global and local information within the modality cannot balance the importance of them, so it is necessary to consider how to adaptively fuse to optimize the feature extraction of the modality. Second, inter- and intra- modality information are complementary, which has been ignored by reported feature fusion methods. It is necessary to consider how to use the complementary inter-modal information to restrict the conditional learning of intra-modal information. In this work, we propose an inter-modal conditional-guided fusion network with Transformer (ICFFormer) to realize adaptive fusion of intra- and inter-modal information and intra-modal feature learning constrained by other modal joint condition information. The experimental results of the clinical hepatocellular carcinoma (HCC) dataset show that the proposed method is superior to the previously reported multimodal fusion methods for HCC grading.