基于深度语法的城市建筑三维模型重建

The main difficulties of urban building 3D modeling lie in the specificity of building structures and the diversity of building styles. The reality, extensiveness and efficiency of current 2D image or 3D point cloud based modeling methods cannot meet the practical demand. Besides, models obtained from current methods usually don’t contain semantic information, which makes the semantic model editing and processing untraceable. To solve the above problems, our project propose a novel framework for modeling semantic building models based on 2D-3D data fusion which better utilizes the deep models’ high capacity of nonlinear fitting and the high accuracy of grammar based modeling methods to improve the performant of building modeling. We first study multi-task based and the 2D-3D fusion based deep models for simultaneous planar segmentation and instance segmentation of urban for urban buildings. Base on this, our project apply a modeling strategy for level of details, modeling the mass model and façade details respectively. For the coarse mass models, our project proposed a hierarchical tree model combined with priori grammar regularities for building structure representation and reconstruction. While for the façade details, based on the deep neural networks and grammar regularities, we study the representation of grammars via deep neural networks, construct the learnable deep grammar model and realize the direct mapping from images to grammars.

城市建筑三维模型重建的主要难点在于建筑结构的特殊性和建筑风格的多样性。当前的基于2D图像或者3D点云的建模方法在真实性、适用性和时效性上尚无法满足实际应用。此外，当前方法得到的模型通常不具有语义信息，不支持语义层面的模型编辑和处理。针对上述问题，本项目研究一种新的基于图像和点云融合的建筑语义模型重建框架，利用深度学习强大的非线性拟合能力，结合语法规则重建的精确性，提升建筑模型重建的效果。本项目首先研究多任务与2D-3D数据融合的深度模型，对建筑同时进行平面提取和实例分割。在此基础之上，采用层次细节的重建策略，分别重建建筑的主体模型和表面细节模型。针对精度要求不高的主体模型，本项目重点研究基于树状分层模型与先验语法规则相结合的建筑模型表达与重建方法。对于表面细节重建，本项目以深度卷积网络和语法规则为基础，研究语法规则的深度表达方法，构建可学习的深度语法模型，实现图像到语法规则的映射。

本项目研究基于图像和点云融合的建筑语义模型重建框架，通过将深度学习和语法规则引入城市建筑的模型重建中，提高模型重建的适应性、真实性和时效性并且重建出具有语义信息的模型。针对项目研究目标，本项目研究了基于多任务和数据融合的深度语义分割模型，通过引入上下文语义信息提升图像语义分割精度。项目中提出了一种基于树状分层模型与语法规则相结合的建筑物表面重建方法，解决传统表面重建方法在大规模三维重建中存在的问题。项目研究了无监督域自适应的立体匹配方法，通过在中间特征空间对合成数据域和真实数据域进行特征对齐，使得特征空间更具有判别性，提升匹配精度。项目还提出了一种三维点云的特征描述方法，通过点云卷积提取高层次点云视觉特征，同时利用密集连接方式和点云语义分割特征融合，提升了特征描述性能。