基于语义分析的大规模场景视觉重建关键技术研究

Three-dimensional model reconstruction technology is widely applied in the national defense, smart city construction, digital culture and many other applications. However, the current three-dimensional fast model reconstruction technology does not make good use of the semantic feature information of the scene, resulting in low efficiency and low accuracy of three-dimensional model reconstruction and other problems. The focus of this research project is developing the theories and methods of three-dimension modeling of large scale scenes based on semantic feature and visual information. This project also focuses on the extraction and expression of the visual feature information and the semantic feature information of the large scale scene image set, tries to utilize these two kinds of information under a unified three-dimensional modeling framework. We solve the segmentation and recognition problems for primitive of different cognitive levels, by machine learning techniques based on huge image data, and solve the illumination reconstruction and the texture reconstruction by extracting the illumination feature of images based on deep learning, and solve the adaptive three-dimensional model reconstruction task of large-scale scene by the fusion of image semantics and geometric semantics, and keep the accuracy and the lightweight of the scene by extracting the semantic structure of the model and using the machine learning techniques. Finally by reconstructing one or two typical complex scene, we verify the modeling quality and efficiency of the fast three-dimensional model reconstruction system which integrates both visual features and semantic features, and provides theoretical and technical support for the efficient three-dimensional model reconstruction of large-scale scenes.

三维建模技术在国防军事、智慧城市和文化娱乐等领域应用十分广泛。但是当前三维快速建模中没有考虑更多的场景语义特征信息，造成三维模型重建效率低、重建精度难以保证等难题，本项目重点研究基于语义和视觉的大场景三维建模的理论和方法，重点关注大规模场景图像集下的视觉特征和场景语义信息的提取与表达，把视觉结构与场景语义统一在一个三维重建框架下实现。通过研究基于海量图像的机器学习技术，解决不同认知层次上的语义基元和对象的分割和识别问题；通过研究基于深度学习技术的图像本征光照特性提取，解决模型的光照重建和纹理重建问题；通过图像语义和模型几何语义的融合，解决大规模场景的自适应重建问题；通过场景模型结构语义的提取和机器学习技术，解决场景模型表示精度和场景数据轻量化问题。并通过1-2个典型复杂场景的重建，验证融合语义和视觉特征的三维快速建模系统建模质量和运行效率，为大场景的高效三维建模提供理论与技术支撑。

三维建模技术在国防军事、智慧城市和文化娱乐等领域应用十分广泛。但是当前三维快速建模中没有考虑更多的场景语义特征信息，造成三维模型重建效率低、重建精度难以保证等难题。本项目重点研究基于语义和视觉的大场景三维建模的理论和方法，重点关注大规模场景图像集下的视觉特征和场景语义信息的提取与表达，把视觉结构与场景语义统一在一个三维重建框架下实现。在不同认知层次上的语义基元和对象的分割和识别、模型的光照重建和纹理重建、大规模场景的自适应重建问题和场景模型表示精度和场景数据轻量化等问题上做出了优秀研究成果。在基于视频或图像的机器学习、视频或图像场景的处理与绘制、大规模场景高效三维重建、三维语义理解和三维结构表达、以及与应用相关（医学仿真、流体仿真）的加速计算等领域发表了高水平期刊或会议论文以及发明专利。研究成果已应用于游戏、城市实景建模、增强现实导航以及无人驾驶高精地图等应用场景。