Speaker Diarization With Region Proposal Network

Speaker diarization is an important pre-processing step for many speech applications, and it aims to solve the "who spoke when" problem. The standard diarization systems can achieve satisfactory results in various scenarios, but they are composed of several independently-optimized modules and cannot deal with the overlapped speech. In this paper, we propose a "nearly" end-to-end speaker diarization system: Region Proposal Network based Speaker Diarization (RPNSD). It can extract individual speech segments even from overlapped speech and compute their speaker embeddings at the same time. Compared with standard diarization systems, RPNSD has a shorter pipeline and can handle the overlapped speech. Experimental results on three diarization datasets reveal that RPNSD achieves remarkable improvements over the state-of-the-art x-vector baseline.