The Good, the Bad, and the Ugly: Neural Networks Straight from JPEG

Over the past decade, convolutional neural networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks. They can learn robust representations of image data by processing RGB pixels. Since image data are often stored in a compressed format, from which JPEG is the most widespread, a preliminary decoding process is demanded. Recently, the design of CNNs for processing JPEG compressed data has gained attention from the research community. They process DCT coefficients instead of RGB pixels, saving computation for decoding JPEG images, however, at the cost of increasing the computational complexity of the network. In this paper, we examine how spatial resolution and JPEG quality impacts on the performance of a state-of-the-art CNN designed to operate directly on the JPEG compressed domain. To alleviate its computational complexity, we propose a Frequency Band Selection (FBS) technique to select the most relevant DCT coefficients before feeding them to the network. Experiments were conducted on a subset of the ImageNet dataset considering both fine- and coarse-grained image classification tasks. Results show that such networks are resilient to JPEG quality but are susceptible to spatial resolution. Also, our FBS can reduce the computational complexity of the network while retaining a similar accuracy.