Dual-Branch Multimodal Fusion Network for Skin Lesions Diagnosis Using Clinical And Ultrasound Image

Skin lesions describe the abnormal skin tissue that may be an indicator of cancer. A variety of skin diseases can be life-threatening. Early diagnosis of benign and malignant diseases is crucial. The use of deep learning-based methods can reduce the influence of personnel subjectivity and improve the accuracy of diagnosis. This paper studies the benign and malignant skin disease diagnosis using a multi-modal fusion network. Specifically, we design a dual-branch learning network. The global branch uses EfficientNet network to extract the feature information of each modality, and the local branch uses convolutional neural network (CNN) to fuse the feature information of another modality, which can interact and fuse the information of the two modalities. We design an attention-based feature fusion strategy to focus more on learning the features of the lesion area, which enhance more discriminative feature information of each modality. Note that this method focuses on learning feature information from clinical images (CN) and grayscale ultrasound images (US). Our method achieves the best performance compared to other methods by conducting experiments on self-collected dataset.