面向多模态信息有效利用的机器学习方法研究

As a large number of multimodal data collected, it becomes desirable to process these data intelligently. To effectively utilize the information contained in multimodal data, we focus on four issues on machine learning techniques while making use of multimodal information. 1. The information from different modals is not with equal importance, i.e., some modals are with rich information while others are poor, and the feature extraction costs are also different for different modals. How to effectively exploit the information form poor modals for fast multimodal learning? 2. Information in different modals is not conditional independent in most case. In order to make modals conditionally independ to each other, how to select or extract features for multimodal data when examples are limited in semi-supervised scenario? 3. Multimodal information can be incomplete for many reasons, e.g., failures of sensors, how to learning with these incomplete multimodal information? 4. How to learning with hidden multimodal data, where the multimodal information is not divided into multiple modals, but hided in a single modal representation, by co-training style approaches? As an application of the solutions to these four issues, we will also build a prototype system. It is expected to publish 8-10 high quality papers on important international journals, conferences and top native journals, apply 2-3 patents, and supervise 5-8 graduate students in this project.

随着大量多模态数据的收集，对多模态数据进行智能处理成为迫切需要。为了有效地利用多模态数据提供的信息，本项目对机器学习利用多模态信息所面临的四个问题进行研究。首先，不同模态所包含的信息存在强弱之分，并且不同模态的特征抽取代价不同，如何利用弱模态信息进行快速、有效的多模态学习？第二，很多情况下，多模态间的信息并非依据类别条件独立，如何在样本极少的半监督环境下对多模态特征进行选择和抽取，尽量使得模态之间的条件独立性得以满足？第三，多模态信息可能因为各种原因而缺失，例如传感设备故障导致模态信息缺失，如何利用不全面的模态信息进行学习？第四，当多模态信息并没有显式地划分为多个模态，而是隐藏在单模态特征中，如何对隐多模态信息加以划分和利用，并使用协同训练类方法加以学习？本项目将为上述问题提供解决方案并研制原型系统，发表国际期刊/会议和国内一级学报论文8-10篇，申请专利2-3项，培养3-5名研究生。

随着大量多模态数据的收集，对多模态数据进行智能处理成为迫切需要。为了有效利用多模态数据提供的信息，本项目对机器学习中利用多模态信息所面临的问题进行研究，提出了能够利用不均衡模态信息的多模态学习方法，提出了能够利用非条件独立模态信息的多模态学习方法，提出了能够利用信息缺失模态信息的多模态学习方法，提出了能够利用隐模态信息的多模态学习方法。本项目为上述问题提供解决方案并研制原型系统，发表国际期刊/会议和国内核心期刊论文共20篇，申请专利2项，培养6名研究生。