Open Set Semantic Segmentation with Statistical Test and Adaptive Threshold

Semantic segmentation in the open world is prerequisite when deploying a well-trained segmentation model in real scenarios, where objects of unseen classes during model training may often appear in future new images to be segmented by the model. However, such open set semantic segmentation task has been rarely explored before. In this study, making use of the large number of pixel-level prediction uncertainties for each image, we proposed applying the non-parametric statistical test to detect whether objects of unseen classes appear in a new image, and an adaptive threshold method to automatically segment each pixel into either one of the known classes or the unknown class. Experiments on the natural image dataset showed that the proposed method significantly outperforms multiple strong baseline methods.