卷积神经网络视觉和语义解释方法研究

The current artificial intelligence based on big data and deep learning has made great progress in the fields of image recognition, speech recognition and natural language processing. However, the deep neural network is essentially a non-linear and non-convex model, with multilayer hidden structure, feature vectorization, massive model parameters, which are difficult for users to understand. So the deep neural network is called as the "black box" model. For the high-risk areas such as military, finance, medicine and transportation, the weak interpretability of the deep learning model becomes the biggest obstacle to the practical application of artificial intelligence. The interpretation methods of convolution neural network(CNN) are the key basic technology to understand and explain the deep network. This project aims at the "black box" problem of CNN, constructs a visual and semantic interpretable framework, and provides users with explanations based on images and natural language descriptions. The interpretation framework of CNN studies the analysis methods of deep visual representation semantic, the construction methods of deep network semantic association, the generation methods of visual interpretation and semantic interpretation, and finally outputs the visual explanation maps with weights and the corresponding natural language explanation descriptions. In order to verify the validity of the interpretation methods, for the typical computer vision tasks such as image classification, an explainable intelligent decision prototype system is built based on the interpretation framework to provide the user with a comprehensive visual and semantic explanation of the decision-making results and processes.

当前以大数据和深度学习为基础的人工智能在图像识别、语音识别、自然语言处理等领域取得了巨大进展。但深度神经网络本质上是非线性非凸模型，存在多层隐藏结构、特征矢量化、海量模型参数等因素让用户难以理解，被称为“黑盒”模型。对于军事、金融、医药、交通等高风险领域，深度学习模型的弱解释性成为人工智能实际应用的最大障碍。卷积神经网络解释方法是理解和解释深度网络的关键基础技术。本项目针对卷积神经网络模型的“黑盒”问题，构建视觉和语义解释框架，向用户提供基于图像和自然语言描述的解释。卷积神经网络解释框架研究深度视觉表征语义分析方法、深度网络语义关联构建方法、视觉解释和语义解释生成方法，最终输出带权重的视觉解释图以及对应的自然语言解释描述。为验证解释方法的有效性，针对图像分类等典型计算机视觉任务，基于解释框架构建可解释的智能决策原型系统，对决策结果和过程向用户提供视觉和语义上的综合解释。

针对当前深度神经网络难以解释和理解问题，本项目聚焦卷积神经网络的解释方法研究。卷积神经网络本质上是非线性非凸模型，存在多层隐藏结构、特征矢量化、海量模型参数等因素让用户难以理解。卷积神经网络解释方法是理解和解释深度网络的关键基础技术。首先，项目构建卷积神经网络模型的视觉和语义解释框架；然后研究深度视觉表征语义分析方法，分析卷积核检测的视觉表征所对应的语义；其次，研究深度网络语义关联构建方法，建立卷积核与单独语义之间的关联关系；进一步，研究视觉解释和语义解释生成方法，生成视觉和语义解释描述；最后，针对典型计算机视觉任务基于解释框架构建可解释的智能决策原型应用。主要结果包括：（1）提出了一种对人类图像分类的视觉语义解构方案和视觉可解释度量方法，用于检测各类典型CNN的内部视觉表征的语义能力；（2）提出了一种用于六边形网格的卷积神经网络HexagonNet，用于处理地理空间信息和智能决策的任务，分类精度达90%以上；（3）构建了用于视觉解释的军事图像数据集，并在此基础上提出了基于视觉解释的深度卷积神经网络迁移学习优化方法，提高了网络分类性能；（4）提出了一种用于视频暴力检测的时间和通道注意力模块，用于检测视频中出现的暴力事件，准确度达到87.5%。本项目研究将为未来图像/视频智能分析、军事智能分析决策解释方面提供新的思路和方法。