Deep Learning for All-in-Focus Imaging

Focus stacking is an effective approach to extending the depth of field of a camera, yet is challenging with regard to 1) controlling focal planes in forming a stack and 2) fusing the focal stack into composites that are free from defocusing, i.e., all-in-focus. We propose an all-in-focus imaging pipeline using deep learning techniques as a novel solution for focus stacking. In the proposed pipeline, three components, namely autofocus, focus control and fusion, incorporate in a closed loop to maximize the quality of all-in-focus estimates. The image-based neural autofocus algorithm acts 5x-10x faster to focus on a single object in contrast to traditional autofocus algorithms based on contrast maximization. Using the neural autofocus algorithm as a tool, the focus control agent based on reinforcement learning can dynamically estimate the environment and plan a focal plane trajectory for a scene with multiple objects. The neural fusion algorithm can better fuse a focal stack through transfer learning. The overall pipeline outperforms traditional focus stacking approaches in both static and dynamic scenes.