Point Set Attention Network for Semantic Segmentation

Self-attention mechanism which aggregates features by capturing long-range relation between each pixel and its context, has been widely used in semantic segmentation task. However, only considering the similarity between pixels is easily disturbed by noisy pixels. In this paper, we propose a Point Set Attention Network (PSANet) for improving self-attention mechanism by correcting the noisy pixels. Specifically, we emphasize to contribute mutual improvement between pixels of the same class. For the pixel at a certain position, we consider current pixel and pixels in its neighborhood as a point set and generate a context-aware mask for selecting pixels belong to the same class as current pixel. Then the selected pixels are aggregated for a center pixel, with which current pixel is updated. In this way, pixels belong to the same class may have similar feature representation, thus promoting intra-class mutual improvement. Next, we compute the relation between each updated pixel and its context features, which often represent all pixels in the feature map. Finally, we aggregate context features on each pixel according to their relations with the pixel. We conduct extensive experiments to validate the effectiveness of our network and achieve outstanding performance on Cityscapes and PASCAL Context datasets.