Semantic-Based Sentence Recognition In Images Using Bimodal Deep Learning

The accuracy of computer vision systems that understand sentences in images with text can be improved when semantic information about the text is utilized. Nonetheless, the semantic coherence within a region of text in natural or document images is typically ignored by state-of-the-art systems, which identify isolated words or interpret text word by word. However, when analyzed together, seemingly isolated words may be easier to recognize. On this basis, we propose a novel ƒ??Semantic-based Sentence Recognitionƒ?� (SSR) deep learning model that reads the text in images with the help of understanding context. SSR consists of a Word Ordering and GroupingAlgorithm (WOGA) to find sentences in images and a Sequence-to-Sequence Recognition Correction (SSRC) model to extract semantic information in these sentences to improve their recognition. To show the effectiveness and generality of SSR in recognizing text, we present experiments with three notably distinct datasets, two of which we created ourselves. They respectively contain scanned catalog images of interior designs and photographs of protesters with hand-written signs. Our results show that SSR statistically significantly outperforms a baseline method that uses state-of-the-art single-word-recognition techniques on these three datasets. By successfully combining both computer vision and natural language processing methodologies, we reveal the important opportunity bi-modal deep learning can provide in addressing a task that was previously considered a single-modality computer vision task.