单目多视角深度图估计的三维目标检测与语义重建研究

3D object detection and semantic reconstruction have been intensively used for vision perception of robot, self or assistant driving systems, robot arm, and VR/AR applications. In this project, we propose a monocular 3D object detection and semantic reconstruction pipeline based on depth estimation from multiple viewpoints, which does not rely on extra 3D sensors, aiming to resolve the challenging task of 3D dense reconstruction from monocular camera. Firstly, we plan to investigate a depth estimation algorithm based on the semantic generative model, which integrates the semantic segmentation with the conditional generative adversarial nets to estimate high resolution depth map. The depth maps from multiple viewpoints are fused to refine the global 3D reconstruction. To reduce the complexity of extracting 3D proposal from 3D point cloud, an optimization method of projecting 3D bounding box to multiple viewpoints is proposed. Moreover, the extracted 3D structure features are further incorporated with 2D appearance and 3D geometric features to improve the 3D detection performance. Lastly, a shape prior based 3D semantic reconstruction method is proposed to segment the object from 3D point cloud, where the object shape is used as constraint in time domain to simultaneously reduce the complexity and improve the accuracy, resulting in the optimized 3D object location and sematic segmentation. This project aims to provide core techniques and theoretical foundations for a variety of relevant computer vision tasks.

三维目标检测与重建能够广泛应用于机器人的视觉感知、自动/辅助驾驶系统、机器手臂的操作、VR/AR等应用中。本项目针对单目视觉难以实现三维稠密重建的瓶颈问题进行研究，提出基于多视角深度图估计的算法，在不依赖于额外三维采集设备情形下实现三维目标检测与语义重建。针对目前单目视觉难以估计高精度的深度图的问题，研究基于语义生成模型的深度图估计算法，以生成高分辨率的深度图；进一步研究基于多视角深度图融合算法，构建动态更新的全局三维重建模型。基于场景的三维重建，研究基于三维包围框的多视角投影优化算法，并融合二维外观特征，三维几何与结构特征，实现三维目标的检测；进一步研究基于形变约束的三维语义重建算法，利用目标形状在时域上的形变约束，降低三维语义分割的优化复杂度，最终得到目标的三维空间位置以及语义分割，为相关视觉感知任务提供关键技术和理论基础。

在国家基金委和学校的支持下，项目组顺利完成了项目申报书中的各项任务，超额完成了申报书中的考核指标和预期研究成果。项目组针对单目视觉难以实现三维稠密重建的瓶颈问题进行研究，在基于语义生成模型的深度图估计、高分辨率稠密三维重建、多特征融合的三维目标检测、语义感知及三维语义重建四个方面进行了研究，取得了较完善的技术研究成果，并将其转化为多项具有现实意义的专利成果，展现了研究成果的实践价值。本项目针对三维目标检测与重建技术在复杂场景中的应用提出了新的研究方法，提出基于自监督学习的深度图估计的算法，在不依赖于额外三维采集设备情形下实现三维目标检测与语义重建，实现了对三维目标检测与重建技术进行了拓展与补充。本项目的研究能够应用于机器人的视觉感知、自动/辅助驾驶系统、机器手臂的抓取、VR/AR等领域。项目组在包括IEEE TIP, TCSVT等国内外知名学术期刊上发表了学术论文12篇，其中SCI收录论文9篇；申请国家专利3项，其中授权发明专利1项，在审发明专利2项。