情感驱动的人机交互中文本语音情感信息耦合关键技术研究

Text and speech is the most direct information carrier in human-machine interaction with the most abundant information, this project analyzed emotional characteristics of text and speech in human-machine conversation basing on coupling relations. Text is treated as bases and speech as supplements, emotional features in text and speech are coupled. At the same time, basing on the coupling of emotional features, emotion transfering in human-machine interaction is extracted and indicated, which sets up the foundation for further research on emotional transfer mechanism of human-computer interaction, realizing nature and harmonious human-machine interaction; Through the analysis of emotional expression integrity in conversation, the studies of text and speech association are put forward on multi-level such as word, phrase and sentence, meanwhile, feature-level text and speech fusion model based on multi-level of emotion features is studied. Multi-granularity and multi-mode emotional information processing has more advantages than single-granularity or single-mode. Basing on coupling emotional characteristics of text and speech, emotional semantic information is adopted and emotional template transfer mechanism in human-machine interaction is studied, then individualized emotional transferring network is constructed and completed to drive the process of human-machine interaction. This study has very important and scientific significance for exploring multi-model emotional information analysis and procesing mechanism in cognitive science, revealing the interaction between various media in the process of human's conversation, and deepening our understanding of how brain performs emotion analysis and understanding.

文本和语音是人机交互中最基础最丰富的信息载体，本项目依据耦合关系对人机交互中的文本和语音情感特性进行分析，以文本为基础、语音为补充，将文本和语音中的情感特征进行耦合，同时基于耦合情感特征对交互中情感转移进行挖掘和表示，为深入研究人机交互中的情感转移机理，实现自然和谐的人机交互奠定基础。通过对交互中情感表达完整性分析，针对文本语音同步关联关系的描述问题，提出在词、短语、句子等多层次进行多粒度文本语音情感耦合，研究基于多粒度的文本语音多模态情感特征级与决策级融合模型，多粒度多模态情感信息处理较单一模态或单一粒度有更大的优越性；基于文本语音耦合情感特征，引入情感语义研究人机交互中情感转移机理，构建完善个性情感转移网络，以情感驱动人机交互。本研究对于对在认知科学上探索大脑对多模态情感信息的分析与处理的机理、揭示人类会话过程中各媒介之间相互作用、进而加深对大脑中情感的解析理解具有重要科学意义。

文本和语音是人机交互中最基础最丰富的信息载体,本项目依据耦合关系对人机交互中的文本和语音情感特性进行分析,以文本为基础、语音为补充,将文本和语音中的情感特征进行耦合,同时基于耦合情感特征对交互中情感转移进行挖掘和表示,为深入研究人机交互中的情感转移机理,实现自然和谐的人机交互奠定基础。通过对交互中情感表达完整性分析,针对文本语音同步关联关系的描述问题,提出在词、短语、句子等多层次进行多粒度文本语音情感耦合,研究基于多粒度的文本语音多模态情感特征级与决策级融合模型,多粒度多模态情感信息处理较单一模态或单一粒度有更大的优越性;基于文本语音耦合情感特征,引入情感语义研究人机交互中情感转移机理,构建完善个性情感转移网络,以情感驱动人机交互。本研究对于对在认知科学上探索大脑对多模态情感信息的分析与处理的机理、揭示人类会话过程中各媒介之间相互作用、进而加深对大脑中情感的解析理解具有重要科学意义。