VMAF Based Rate-Distortion Optimization for Video Coding

Video Multi-method Assessment Fusion (VMAF) is a machine-learning based video quality metric. It is experimentally shown to provide higher correlation with human visual system as compared to conventional metrics like peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) in many scenarios and has drawn considerable interest as an alternative metric to evaluate the perceptual quality. This work proposes a systematic approach to improve the video compression performance in VMAF. It is composed of multiple components including a preprocessing stage with a complement automatic filter parameter selection, and a modified rate-distortion optimization framework tailored for VMAF metric. The proposed scheme achieves on average 37% BD-rate reduction in VMAF, as compared to conventional video codec optimized for PSNR.