基于结构化深度学习的场景理解

In this proposal, we aim to systematically study on structured deep learning based scene understanding. We propose to design proper deep neural networks, in order to overcome the shortcomings of the traditional models on the lack of ability on image feature representation. We integrate deep neural networks and probabilistic graphical models to study the structured deep learning problem. we propose a model which combines convolutional neural networks (CNN) and continuous conditional random fields (CCRF) in a unified deep learning framework. CNN can extract rich related features from image, and CCRF can optimize the output of CNN according to the position and color information of the image pixels. We also propose a model to fuse sparse known labels. By using these known labels, we can provide our model a few relatively accurate depth value as a reference, and largely narrow down the range of reasonable depth value of other pixels. Thus the model can reduce the ambiguity of mapping a single RGB image into depth map to some extent. We investigate the multi-task learning problem on both depth estimation and semantic segmentation. By addressing the problems studied in this proposal, the vision based scene understanding theory and technique can be further developed.

本项目旨在研究基于结构化深度学习的图像场景理解问题，通过合理设计和架构深度学习网络模型，克服传统方法在图像特征表达能力方面的不足；将深度学习模型与概率图模型相结合，研究统一的结构化深度学习框架，解决传统深度学习模型输出缺乏结构化约束、空间结构信息丢失的问题；面向大规模应用数据，研究结构化输出模型的快速推断方法，以避免巨大的计算代价；在深度学习模型中引入可以通过较低成本获取的稀疏先验信息，研究先验信息的融合方法，以减少单幅图像到深度图之间映射的不确定性。研究图像的深度估计和语义分割的多任务学习，实现两个任务的共同增强。本项目的研究将促进基于视觉的场景理解理论和技术的发展，为机器人、无人车等技术的推进和发展提供有力的保障。

本项目深入研究了基于结构化深度学习的图像场景理解问题。围绕场景语义描述、语义分割、深度估计等计算机视觉关键科学问题展开系统研究，提出了新颖的方法，取得了一定的成果，包括：面向场景分析的自适应上下文神经网络模型、基于反事实多智能体强化学习的非自回归场景语义描述模型、基于多视图特征与混合奖励的场景语义描述方法、基于双重关系感知注意网络的场景解析方法、基于连续条件随机场的单目图像场景深度估计神经网络等。项目组共发表论文11篇，其中发表期刊论文4篇，发表高水平国际会议论文7篇。本项目的研究将促进基于视觉的场景理解理论和技术的发展，为机器人、无人车等技术的推进和发展提供有力的保障。