基于言语感知机理的深度学习方法的研究

While building a deep neural network with the traditional acoustic features and the multimodal speech production features are used, which makes the feature maps of the deep architecture have a physical meaning, is an interesting topic both in theory and application, which could widely utilized speech recognition, artificial intelligence, and so on. With a perspective of speech chain, on the base of brain mechanisms of speech perception, using the map relationship between articulation and acoustic space, this project try to fusion of the acoustic and multi-modal articulation features in the deep architecture. Compare with traditional acoustic features in deep neural network, those new features will correspond to more specific physical quantity, such as manner of articulation or place of articulation. This research utilize multimodal speech production database to build a map between acoustic and speech production features firstly, then build a multi-modal deep neural network model, at the end a multi-modal convolutional deep neural network is built, in which the convolutional kernels are trained by the using of acoustic or multi-modal articulation features. In the application, the deep neural network model is more robustness and is easier to be used in a multi-tasking system. In theory, this research will also further promote the theory and application of deep Learning in the field of speech recognition.

在利用传统的声学特征进行深度神经网络建模的同时融入语音发音等多模态特征，使深度结构中各层特征对应实际物理意义，是颇具理论和实际意义的一项科研课题，并且在语音识别，人工智能等领域有极其重大的应用价值。本课题从言语链的角度出发，结合人脑的言语感知机理，利用语音发音和声学空间的映射关系，试图在深度结构中进行多模态发音特征和声学特征的融合。与传统深度神经网络的语音声学特征相比，这些发音特征将对应更确切的物理意义，例如: 发音部位，发音方法等。本项目首先利用多模态数据库建立声学和语音生成特征映射关系，然后建立多模态深度神经网络模型，最后利用声学、多模态发音数据训练卷积核，构建多模态深度卷积神经网络。在应用方面,由于该深度神经网络模型有效地融入了人的语音处理功能，有望更好地解决鲁棒性问题。在理论方面,这项研究也将进一步为推动深度学习技术在语音识别领域的应用提供理论支撑。

本项目开展了利用传统的声学经验在深度神经网络建模的同时进行发音等传统特征学习方法的研究，旨在使深度结构中各层特征对应实际物理意义。项目组从言语链的角度出发，结合人脑的言语感知机理，利用语音发音和声学空间的映射关系，在深度结构中进行多模态发音特征和声学特征的融合。与传统深度神经网络的语音声学特征相比，这些发音特征将对应更确切的物理意义，例如: 发音部位，发音方法等。利用睡眠鼾声等数据库建立声学和语音生成特征映射关系，建立了声学睡眠监测深度神经网络模型，在语音传统经验和深度网络建模结合提出了一种思路。在应用方面,由于该深度神经网络模型有效结合人机交互、医疗康复等应用，更好地解决智能识别系统鲁棒性问题。