Adversarial Deep Features for Weakly Supervised Document Image Keyword Spotting

Keyword spotting (KWS) aims to retrieve all instances of particular keywords in a document. Modern approaches exploit the representational power of convolutional networks (CNN) to produce discriminative word image representations able to perform in challenging multi-writer conditions. However, they require lots of training data while their adaptivity to unknown document collections when little or no annotations exist is uncertain. To this end, we utilize a CNN to extract intermediate layer deep features. We then combine adversarial learning with spatial transformer networks to obtain discriminative deformations of the feature space leading to compact deep feature representations which alleviate the adaptation of the proposed KWS system into weakly supervised manuscripts. Numerical experiments of the adaptation of deep features from a low resource document collection (GW) to a much more diverse target dataset (IAM) where little annotations exist to fine-tune the original model are on par with the state of the art.