Compressed Video Sensing Network based on Alignment Prediction and Residual Reconstruction

Image compressed sensing networks based on deep learning have achieved promising performance recently, which provides a new idea for Compressed Video Sensing (CVS). For CVS, the reconstruction performance can be improved by utilizing the correlation between adjacent frames. Following the traditional ‘prediction-residual reconstruction’ framework, we propose a novel learnable network for CVS based on alignment prediction and residual reconstruction named PRCVSNet. In PRCVSNet, Temporal Deformable Alignment Network (TDAN) is introduced to predict the current frame from reference frames for the first attempt. Furthermore, we design a novel residual reconstruction mechanism which can be applied in both initial and enhancement reconstruction stages to improve the reconstruction performance of video with the measurements. At last, the PRCVSNet introduces a fusion network taking advantage of information of multiple reference frames. Experimental results demonstrate the effectiveness of our network against the state-of-art methods with improvement of 1-5 dB in PSNR.