异构媒体深度分析与跨领域理解

Multimedia big data is becoming more socialized and isomerized. Considering such inevitable trends, we propose a set of computational theories for deep analysis of heterogeneous multimedia data, which can direct the further exploration of the scientific problems of multimedia content analysis and understanding under the background of the big data era. This project starts with the investigation of the inner correlations within heterogeneous multimedia data, trying to construct a theoretical framework for hierarchical content parsing, cross-domain feature representation and weakly supervised lingual-visual embedding. We reveal the semantic correlations and the statistical characteristics existed in the socialized and isomerized multimedia data, and investigate the co-analysis and joint modeling strategies for both visual and lingual contents. By fusing the spatio-temporal features and the contextual constraints from both visual and lingual data, we could achieve large-scale and fine-grained lingual-visual embedding, serving as the theoretical foundations for heterogeneous multimedia analysis. We discover close correspondences between visual contents and lingual fragments, and further establish a unified representation for both visual and lingual data in the embedding space, on top of which we can achieve technical breakthroughs such as multimedia retrieval with complex lingual queries as well as knowledge visualization and descriptions. The final achievement of this project will be a unified system for the analysis and deep understanding of heterogeneous multimedia contents, and also a wide range of applications in the domain of both industry and education.

针对多媒体大数据的社会化、异构化趋势，建立一套面向异构媒体数据深层次分析的计算理论，指引探索和解决大数据背景下多媒体内容分析与理解中存在的科学问题。本项目将探索异构多媒体数据的内在关联特性，建立视觉多媒体数据的层次化解析、异构关联数据的跨领域特征表达、弱监督的视觉 - 语言嵌入学习等理论框架，揭示社会化、异构化媒体数据中泛在的语义关联和统计特性，探索视觉内容与语言内容的协同分析和联合建模策略，将视觉与语言各自的时空特性和上下文约束结合起来，逐步构建面向异构大媒体分析的计算模型，实现大规模、细粒度的视语嵌入学习，建立视觉内容与语言片段的紧凑关联，得到视觉内容与语言内容在嵌入空间中的统一表示，进而突破面向复杂口语化查询的多媒体信息检索，及知识的可视化和口语化描述等关键技术，最终实现异构大媒体数据的深度分析与理解，及其在工业、教育等领域的应用。

项目探索了异构多媒体数据的内在关联特性，建立视觉多媒体数据的层次化解析、异构关联数据的跨领域特征表达、弱监督的视觉 - 语言嵌入学习等理论框架，揭示了社会化、异构化媒体数据中泛在的语义关联和统计特性，探索视觉内容与语言内容的协同分析和联合建模策略，将视觉与语言各自的时空特性和上下文约束结合起来，逐步构建面向异构大媒体分析的计算模型，实现大规模、细粒度的视语嵌入学习，建立视觉内容与语言片段的紧凑关联，得到视觉内容与语言内容在嵌入空间中的统一表示，进而突破面向复杂口语化查询的多媒体信息检索，及知识的可视化和口语化描述等关键技术。课题组发表学术论文20篇，包含IJCV等10篇SCI期刊和CVPR、NeurIPS等10篇CCF-A类国际会议。