多模态融合的视频监控数据交互式查询及分析研究

Rapid advances in Internet of Things, cloud computing, and intellisense devices have led us into an era of ubiquitous intellisense. Given the massive video surveillance data, it has become an essential issue to manage and utilize the information, effectively and efficiently extract semantics, in order to support a wider range of intelligent analysis and knowledge discovery applications. However, the traditional video intelligent analysis system has limited computing power, low robustness and poor interaction, unsatisfying the requirements of practical applications. Therefore, in this proposal, we target to address the key issues for interactive query and analysis of video surveillance data based on multi-modality fusion. Given the acquired massive video data, we first propose to mine and select the effective muli-modality features. We then construct entity relationship graph based on these features, to mine the correlation between entities and extract semantic information employing probabilistic graphical model. Based on the extracted semantics, we next design the unified data model, and construct the big video database and semantic probabilistic database leveraging the big data management system (AsterixDB). With the database, we provide interactive query function to help users achieve the accurate results as soon as possible. Our research will provide technical support for researches on intelligent analysis and knowledge mining of big video surveillance data, and meanwhile make important advance to the construction and development of smart community and smart city.

物联网、云计算、以及智能监控设备的迅猛发展把我们带入了一个智能感知无处不在的时代。面对海量视频监控数据，如何加以管理与利用，准确高效的提取语义信息以支持更为广泛的智能分析与知识挖掘成为至关重要的问题。然而，传统的视频智能分析系统计算能力有限、鲁棒性较低，交互性较差，无法满足实际应用需求。为此，本课题致力于研究多模态融合的视频监控数据交互式查询及分析关键技术。针对采集的海量视频数据，本课题拟首先挖掘并选择有效的多模态特征；然后基于此特征构建实体关系图，利用概率图模型算法挖掘关联关系进而提取语义信息；基于得到的语义信息，设计统一的数据模型，利用大数据管理系统（AsterixDB）构建海量视频数据库和语义概率数据库；最后基于此数据库，提供交互式查询功能，帮助用户尽快得到准确的查询结果。本项目可为海量视频监控数据智能分析与知识挖掘提供有力的技术支撑，也将有力推进智慧社区、智慧城市的建设和发展。

在智能感知无处不在的时代，如何管理与利用海量视频监控数据，准确高效的提取语义信息以支持更为广泛的智能分析与知识挖掘成为至关重要的问题。传统的视频智能分析系统计算能力有限、鲁棒性较低，交互性较差，无法满足实际应用需求。本课题致力于研究多模态融合的视频监控数据交互式查询及分析关键技术。针对采集的海量视频数据，我们首先进行前景目标识别，并挖掘有效的多模态特征；然后构建实体关系图挖掘关联关系进而提取语义信息；基于得到的语义信息，设计统一的数据模型，利用大数据管理系统构建海量视频数据库；最后基于此数据库，提供交互式查询功能，帮助用户尽快得到准确的查询结果。相关研究成果已经在多媒体数据分析领域的著名国际会议及期刊发表论文22篇，其中SCI检索论文11篇（包括CCF A类期刊IEEE T-PAMI 3篇，IJCV 2篇，CCF B类期刊IEEE TNNLS 2篇， TOMM 1篇），会议论文11篇（包括CVPR，AAAI等CCF A类会议2篇），并获得CCF C类会议MM Asia 2020最佳论文奖。