DDN: Dynamic Aggregation Enhanced Dual-stream Network for Medical Image Classification

Convolutional Neural Networks (CNNs) have become the de facto approach for medical image classification in recent years. However, the deficiency of convolutional operations in extracting global features has limited the further improvement of this task. Vision Transformers (ViTs) can model long-range dependencies via self-attention mechanism but unfortunately lose local feature details. In this paper, we propose a dynamic aggregation enhanced dual-stream network termed DDN to take the advantage of ViT and CNN to enrich the feature representation of medical images. Specifically, our proposed DDN is built by stacking several Dynamic Dual-stream Units (DDU). In DDU, local and global features are learned by the CNN branch and Transformer branch respectively whilst complementing each other via a bi-directional propagation strategy, then features of both branches are aggregated in a dynamic manner and the integrated information is used to enhance the feature representations of the two branches simultaneously. Extensive experiments show that our proposed DDN performs best compared with other state-of-the-art models on the public Kvasir dataset and ISIC2018 dataset.