Learning-Based End-To-End Video Compression With Spatial-Temporal Adaptation

To solve the low spatial and temporal resolution issue in the conventional hyperspectral (HS) imaging sensors, coded aperture snapshot HS imaging, which encodes the 3D HS image into a 2D compressive snapshot and then adopts computational technique to recover the latent HS image, has attracted remarkable attention in recent year. This study aims to reconstruct the latent HS image with the detail spectral distribution from its compressive snapshot using the deep convolution neural network (DCNN). We leverage an auxiliary RGB learning task to reconstruct the corresponding RGB image from the snapshot image in training phase, and incorporate the learned features of the auxiliary task to assist the more difficult reconstruction of the latent HS image. Specifically, we design the DCNN architecture with two branches for both reconstruction learnings of a small number of spectral image (such as RGB) and the full-spectral HS image, and then integrate the intermediate features in the auxiliary RGB branch to the HS reconstruction branch for augment the spectral learning capability. Experimental results demonstrate our proposed method with the auxiliary learning can achieve comparable performance with the state-of-the-art methods while enable the reduction of the model size.