基于多维特征分析的音画情感联觉检索研究

There is rich emotional information within muli-media data, and "emotion" has become an important index term in retrieval. At present, user's emotional needs in retrieval is becoming more and more prominent. Currently semantic term is mainly applied in the retrieval, however it has some essential shortcomings, that semantic could not fully convey the multidimensional and fuzzy character of emotions, and the recall ratio and accuracy could hardly meet users' requirement..In order to solve the problems of sematic limitations on emotional needs in retrieval, we try to find new way of emotion synesthesia retrieval method. The emotion features will be applied as a bridge to link the music data and image data based on affective computing theory and methodology, a cross-media retrieval method will be created to meet user needs, and we attempt to find a method of affective computing which is suitable for computer to realize computer emotional synesthesia..Three database will be built in the research, including a music emotion feature database, an image emotion feature database and a physiology emotion feature database. Deep learning algorithms will be applied in the emotion modeling work. At last, a music-image cross-media emotion retrieval system will be made based on music-image emotion synethesia space, and computer music-image emotional synesthesia intelligence will be initially realized..

多媒体数据中蕴含丰富的情感信息，“情感”已经成为检索多媒体数据的重要方法，用户检索的情感需求日益突出。目前的检索应用主要采用语义检索技术，其本质上存在瓶颈，即语义很难涵盖情感的多维化与模糊性，导致其查全率和准确率已很难满足用户要求。.本项目针对以上问题，力图研究一种新型的情感联觉检索方式，解决语义在情感需求上的局限性。研究以情感特征为桥梁，基于情感计算理论与方法，创造出接近用户情感需求的音画跨媒体检索方式，试图找出一种适合计算机计算的情感计算规则来实现计算机音画情感联觉。 .研究将构建音乐情感数据库、图像情感数据库及生理情感数据库，然后运用深度学习算法进行多维特征情感识别建模，最后基于音画情感联觉空间，研发一个音画跨媒体情感检索系统，初步实现计算机音画情感联觉。

鉴于“情感”已经成为检索多媒体数据的重要方法，同时也是重要的用户检索需求，本项目研究提出了一种基于情感相似性的情感联觉检索方式，以解决语义在情感需求上的局限性，力求创造出接近用户情感需求的音画跨媒体检索方式。.本项目研究期间构建了相对完善的音乐情感数据库、图像情感数据库及生理情感数据库，重点基于多维特征数据库比对了多种机器学习算法的情感识别模型表现。在多维情感特征的交叉组合中，我们发现包含多媒体特征及用户实时生理特征的特征组合在识别中可以达到最优表现，基于最优模型建立了音画跨媒体检索系统，研发了多个情感交互相关应用系统，初步实现计算机音画情感联觉，并将研究延伸至教育情感智能领域。