Interactive Low Latency Video Streaming Of Volumetric Content

Low latency video streaming of volumetric content is an emerging technology to enable immersive media experiences on mobile devices. Unlike 3DoF scenarios where users are restricted to changes of their head orientation at a single position, volumetric content allows users to move freely within the scene in 6DoF. Although the processing power of mobile devices has increased considerably, streaming volumetric content directly to such devices is still challenging. High-quality volumetric content requires significant data rate and network bandwidth. Although compression algorithms for volumetric video are being developed, no energy efficient hardware for decoding of volumetric data such as point clouds or meshes is available on existing mobile platforms. Moreover, software decoding may not be able to meet real-time processing requirements and increases the battery consumption. Our demonstration mitigates the above problem by offloading the rendering to a dedicated server at the network edge and avoids transmission of the original volumetric content to a mobile client. One major challenge of such cloud rendering is to meet end-to-end latency requirements and ensure a smooth user experience. Typically, placing the rendering engine at a 5G edge server can drastically reduce the network latency. The demonstration setup consists of a browser-based client that displays the 2D video containing the volumetric content and transmits user interactions to the rendering server through WebRTC data channels or websockets. On server side, a rendering engine generates an appropriate view of the volumetric content based on the user interaction. A fast GPU-based video encoder then compresses the rendered 2D view and transmits it to the client using the WebRTC protocol. The described system achieves a very low end-to-end latency, transmits the volumetric content at low bitrates and is suitable for deployment in thin client environments.