基于多视角特征相关性挖掘的大规模异构媒体融合标注方法研究

In this study, we aim to provide an integrated annotation framework for heterogeneous media data, which can annotate object across all of the media types (text, image, audio, video, and 3D model). Multimedia annotation has become a key problem in both research and application. However, existing methods usually focus on modeling a single media type or pairwise correlation. In fact, heterogeneous media are complementary to each other and optimizing them simultaneously can further improve accuracy. First, a unified multi-view feature learning and content representation framework for heterogeneous low-level features is presented by explicitly modeling the relevance between content and labels. And object-label correlation is reconstructed in a nonlinear manner by space projection, and media labels can be retrieved with pair-wise similarity. Since correlation of different media type is complementary to each other and optimizing them simultaneously can further improve the accuracy, a shared subspace learning method with intrinsic structure preservation and semi-supervised regularization is presented to explore the correlation in a joint optimization framework, by which different media can align with each other and thus, which is robust to the noise. Meanwhile, the extension mechanism of the method is also presented to improve its generalization ability. This study also presents a contextual information diffusion and evolutionary model in the shared subspace to improving annotation accuracy. Learning approach to incorporate diversity and relevance for labels in the shared subspace is also presented, to solve such problems which were due to ambiguity, redundancy and the lack of diversity. Proposed methods should be both spatially and temporally efficient to handle large-scale datasets. The results of this study can be applied to multimedia understanding retrieval and mining, by which system performance will improve significantly with a prosperous application.

本课题是研究融合图像、音频、视频、三维模型等异构媒体进行联合语义标注的新课题。语义标注是多媒体检索与挖掘的重要支撑技术，但现有成果只是标注单一类型媒体或对两类媒体进行相关性分析，无法完整、准确地给出语义描述。通过直接对内容特征和文本特征的相关性进行建模的方法，获取多视角特征的一致性描述，建立语义空间的非线性映射模型，将媒体标注问题转换为近邻搜索问题。研究保持异构媒体语义关联性及空间本征结构的异构媒体共享子空间半监督构建方法，并给出模型的更新和扩展机制，使得不同类型媒体数据能够关联和互补，提高模型准确性、健壮性和泛化能力。设计语境信息快速传播和演化机制，实现子空间中深层次语义关联信息的挖掘。并提出子空间中语义相关性和多样性保持方法，解决子空间内的弱标签和语义单一化问题。所提出模型和方法具备规模化处理能力。研究成果可大幅提升标注性能，为媒体理解与挖掘等应用提供新的解决方案，具有广阔应用前景。

本项目研究目标为融合异构媒体进行联合语义标注。本课题从以下三个方面开展了研究工作：在研究的第一方面，本项目进行了媒体多视角特征相关性挖掘的研究。提出了基于用户生成标签的多视角特征学习方法，利用媒体的社会化标签和其内容特征得到优化后的特征表示。通过媒体词汇和文本词汇的相关性建模，学习文本特征空间和内容特征空间的映射模型。给出了优化前后的特征表示具备近似等距映射保持的理论依据。该方法相对数据集规模具备线性时间复杂度，并具备多视角特征融合能力。此外，我们提出了一种多媒体同构空间构造方法。空间变换过程采用正交投影变换，通过非线性方式重构“多媒体-标签”相关性，并保证同构空间与内容特征空间和标签向量空间分别具备结构保持。数据集上的测试表明，所提出的方法取得了更好的标注性能。在研究的第二方面，本项目进行了异构媒体的语义关联性挖掘的研究。我们提出了异构媒体融合空间构建方法，该方法能够共同学习出不同媒体类型的稀疏特征映射矩阵，使得不同媒体类型之间能够互相促进。此外，我们提出了一种语境信息传播和进化模型。构建初始标签语境相关图和图像内容相关图，将该问题描述为两个无向加权图上的损失函数最小化问题。采用数据分割，逐次优化和放松约束的策略，获得该问题的近似解。实验结果表明上述方法具备近似线性的时间复杂度。在研究的第三方面，本项目进行了媒体标签重要性选择机制的研究。我们提出了标签相关性保持方法。该方法给出了标签集对媒体相关性和标签集内部一致性的估计方法，进而采用贪心搜索策略获取近似最优解。此外，我们定义了一个融合相关性和多样性的度量，进而提出了一种标签多样性保持方法，通过启发式的迭代求解过程得到兼具相关性和多样性的标签集合。数据集上的实验结果表明，上述方法可以涵盖更广泛的语义概念。