光场图像新视点编辑算法研究

With the quickly maturation and development of light field cameras, light field imaging has attracted great attention from academia and industry. Light field imaging technology is believed to be the main direction for digital development, and will lead revolution in digital photography and image processing. This proposal application attends to focus on new view synthesis and editing for light field images. It mainly contains three issues/problems as follows. First, we study novel methods for new perspective views generation from light field images. We will propose effective and efficient algorithms to generate high-quality novel views in arbitrary viewpoint position and rotation angle, which also supports super-resolution effect. Since the image quality of synthesized novel views always heavily relies upon the accuracy of disparity maps of all subaperture views of light field images, we will try to estimate accurate disparity maps for all subaperture view and collaboratively optimize the disparity maps for multiple light field images. Second, we study image retargeting methods for light field images. We will propose novel methods to edit the key parameters of the light field image, such as the baseline between the sub-viewpoints and the focal distance etc., and generate the target light field image once the key parameters have been modified. In addition, we will extend our method to support light field image stitching and panoramic image generation, object copy and paste between different light field images etc.. Third, we study methods to remove foreground occlusions from light field images and restore/optimization the background images. Our proposed methods will automatically detect and segment the foreground occlusions, such as mesh, branches, raindrops, snowflakes, and shadows reflected glass window. In addition, our methods will be able to restore and optimize the background light field images.

随着商业光场相机的成熟和推广，光场成像技术引起了学术界和工业界极大的研究关注。光场成像技术被认为是未来数字图像技术发展的主要方向，将引领数字摄影和数字图像处理的革命。本课题围绕光场图像的新视点编辑，重点研究以下三个问题。一是基于光场图像的新视点生成方法。研究基于光场图像生成任意视点位置和旋转角度高质量透视图像的算法，支持图像超分辨率。因为新视点图像的质量依赖于子视点视差图的精度，本课题计算所有子视点精确的视差图，多张光场图像视差图的协同优化。二是光场图像的重定向方法。研究编辑光场图像的关键参数，如子视点间的轴距和焦平面的距离等，生成关键参数修改以后的目标光场图像的算法。进一步研究支持光场图像拼接和全景图像生成，物体拷贝和复制等光场图像编辑任务。三是光场图像前景遮挡物的自动去除方法。研究诸如雨水，雪花，玻璃窗反射的影子等前景遮挡物的自动检测和分割，移除前景遮挡物后背景图像的修复和优化算法。

近十年来，光场图像处理技术引起了学术界和工业界极大的研究关注。光场处理作为AR，VR，MR和元宇宙等领域非常重要的数字内容来源，重要性日益凸显。光场成像和光场图像处理技术仍然最有希望引领数字摄影和数字图像处理技术的革命。本项目围绕光场图像的新视点编辑，重点研究了以下三个方面的问题。一是高效的光场图像的新视点生成算法研究。我们比较系统地研究了在光场图像上实现视点插值情况的新视图生成，包括带有透视效果的双目立体新视图生成算法，超分辨率算法和光场图像暗光增强的算法算法，代表性的研究成果“Stereoscopic Image Generation From Light Field With Disparity Scaling and Super-Resolution”发表在IEEE TIP（TOP期刊）。二是关于光场图像重定向的算法研究了光场图像基线编辑情况下的光场图像重构（生成）算法，光场相机位置发生移动且姿态发生旋转情况下的光场图像透视变换算法，提出了两个新算法。我们还提出一种简单而有效的基于软聚类的全局边缘保持滤波器，并提出了一个基于受限高斯混合模型的软聚类算法来实现图像保边滤波，研究成果“Edge-Preserving Image Filtering Based on Soft Clustering”发表在IEEE TCSVT。三是基于多视图内容分析的光场图像前景遮挡物检测和去除算法。围绕光场图像去雨条纹，去雪花，研究提出了4个新算法。代表性的工作“Rain Removal From Light Field Images With 4D Convolution and Multi-Scale Gaussian Process”，“Rain Streak Removal From Light Field Images”，“Snow Removal From Light Field Images”分别发表在IEEE TIP，IEEE TCSVT, IEEE ACCESS等学术期刊上。另外，我们一个最为重要的研究工作，“GhostingNet: a Novel Approach for Glass Surface Detection with Ghosting Cues”，进入到IEEE TPAMI期刊二审。项目支持培养已毕业的6名硕士研究生，申请国家发明专利6项。