real-time Human reconstruction based on human pose prior and epipolar refinement

Current vision-based human-robot collaboration systems use a human skeleton model instead of human physical boundary for collision-free trajectory planning, which could lead to high security risk in applications. In this paper, we propose a real-time human performance capture framework to generate high-quality surface geometry from multi-view RGB inputs. To achieve run-time efficiency and robust reconstruction, we decompose the human pose prior to human keypoint detection, joint radius estimation as well as human segmentation and fuse them into our multi-task model YOLO-DKS to estimate parameters for coarse reconstruction. After that, we exploit the coarse-to-fine reconstruction paradigm, starting with coarsely reconstructing human physical boundary and following by refinement using masks and RGB images from different viewpoints to obtain accurate surface topology. Experiments demonstrate the effectiveness of our approach in achieving high-quality geometry and efficient computation in comparison with state-of-the-art methods.