基于目标语义的网络图像检索技术研究

With the advance of digital cameras and high quality mobile devices as well as the internet technologies, there are increasingly large amounts of images available on the web, which necessitates effective and efficient image retrieval techniques. Among them, object semantics based image retrieval (OSIR) becomes a newly-developing and potential direction to meet users' high requirement for precise image searching, which is our focus in this proposal. Different from traditional content based image retrieval, the main target of OSIR is to provide object-centric image retrieval framework, in which we should understand image semantics and user query intents from the object level. That is, we should find out the objects included in each image, and provide relevant images on object semantics to users. To achieve this target, we propose to carry out the framework from the following four aspects. First, we will study the characteristics of consistency and complementary among multi-modal image features, based on which we propose effective solutions for feature fusion and representation. Second, the problem of image semantic parsing will be explored with a unified framework, in which image-level annotation, region-level annotation (i.e., object semantic understanding), and image segmentation are jointly explored. Third, in order to achieve an efficient and effective image retrieval system, we will attempt to construct an object-centric image indexing structure with consideration of complex correlations among images on social websites. Finally, how to learn users' query intent and further to rank images based on object semantics will be deeply studied.

基于目标语义的网络图像检索是在新一代互联网背景下有效满足用户精准搜索需求的发展必然，同时也面临着数据规模巨大、噪音信息充斥、索引与搜索粒度精细等现实挑战。在此背景下，本项目将围绕基于目标语义的网络搜索相关技术展开研究，力图通过计算手段实现针对图像目标对象的细粒度语义理解，并基于此构建适于反映图像所包含目标对象的特征索引与搜索排序算法，准确捕捉用户的查询意图，以便提供更加友好、快捷、准确的网络图像资源检索服务。具体拟将从以下四方面展开研究：（1）基于多模态特征之间相容互补特性的图像特征融合与表示；（2）将图像整体标注、区域标注和区域分割相统一的基于弱监督学习的目标语义解析计算框架；（3）联合考虑网络媒体复杂关联与目标语义指导的图像特征索引构建；（4）基于相关反馈的用户查询意图推断与搜索排序算法。

基于目标语义的网络图像检索是在新一代互联网背景下有效满足用户精准搜索需求的发展必然，同时也面临着数据规模巨大、噪音信息充斥、索引与搜索粒度精细等现实挑战。在此背景下，本项目将围绕基于目标语义的网络搜索相关技术展开研究，力图通过计算手段实现针对图像目标对象的细粒度语义理解，并基于此构建适于反映图像所包含目标对象的特征索引与搜索排序算法，准确捕捉用户的查询意图，以便提供更加友好、快捷、准确的网络图像资源检索服务。通过近四年的深入研究与探讨，主要在以下四方面取得了一定的技术突破与研究成果：（1）基于多模态特征之间相容互补特性的图像特征融合与表示；（2）基于反卷积神经网络的目标语义解析计算框架；（3）融合目标语义的判别性图像特征索引构建；（4）基于目标对象的图像排序搜索。课题整体进展顺利，已圆满完成预定目标。相关成果获得中国图像图形学会科学技术二等奖，共发表高水平学术论文21篇（含2篇已接收），其中包括TNNLS、TIP、TKDE等的 SCI索引国际期刊论文9篇，包括ICDE、ACM MM、ICIP等的EI索引国际会议论文12篇，有1篇论文获得ACCV 2018最佳应用论文提名奖，并申请或已授权国家发明专利3项。取得上述创新性研究成果的同时，也注重科研成果的落地应用，研制了一套基于安卓平台的交互式目标语义图像检索系统原型，可提供面向购物场景的实时准确的图像检索服务。核心技术成果也多次在国际大赛中取得优异成绩，其中在ICCV 2017图像场景解析挑战任务中，击败谷歌、今日头条、亚马逊、牛津大学、清华大学等国内外著名AI实验室以及知名科研院校，获得最终冠军；在 2018年由京东公司承办的时尚单品检索任务中战胜100+支队伍取得冠军。