A multimodal fusion framework for brand recognition from product image and context

The detection and recognition of a brand from product images is a key capability in many computer vision and machine learning applications. Specifically, logo detection from image is one of the most distinctive and effective ways to determine the brand. However, due to the large variation in scale, geometry and appearance, etc., logo detection and recognition remains a challenging problem, even with the recent advances using deep neural networks. Another informative source for brand recognition is the textual information within the context as in most of the e commerce websites, a product picture is often accompanied by some text description as well. To combine the best of both worlds, we propose to tackle brand recognition with a multimodal fusion framework that integrates image-based logo recognition using convolutional neural networks with context feature (product image title, description, OCR text detection from image, etc.)-based brand recognition using natural language understanding models. We demonstrated experimentally that the additional context information has significantly mitigated the limitations experienced by image-only-based logo recognition. It is worth noting that, in order to better represent text within its context, we have adopted the texts embedded using BERT in our framework.