基于多示例学习和半监督学习的手势语识别的研究

Sign language recognition is a research focus in the pattern recognition and the intelligent human-computer interaction fields, and is of great theoretical significance and practical value. For vision based sign language recognition, the performance decreases obviousely in the case of cluttered dynamic backgrounds, view and signer variances, which limits the progress of sign language recognition research. At the same time, the multimodal characteristic of sign language is not mined sufficiently in the previous research. This project aims to solve these problems. To solve the problem of recognition in the clutteredd dynamic backgrounds, we introduce the multi-instance learning to sign language recognition, which can recognize gestures even when the hands are not well located and tracked. For multimodal sign lanuage recognition, we add the lip movement, facial expression and head pose information to the multimodal multi-instance framework. To recognize sign language in the cases of view variance and signer variance, we use large amount of unlabeled data to tailor the parameters of the models with the scheme of semi-supervised learning.

手势语识别是模式识别和智能人机交互领域的研究热点，具有重要的理论意义和应用价值。基于视觉的手势语识别中，在复杂动态背景、视角变换和用户变换情况下其系统性能下降明显，已成为阻碍手势语识别研究发展的瓶颈问题；同时手势语的多模态特性也未能受到足够重视；本项目在以往相关研究的基础上对这些问题逐一进行研究。针对复杂动态背景下的手势语识别问题，研究基于多示例学习框架的手势语识别方法，以确保系统在手部区域定位和跟踪失准时仍能正常识别；针对手势语的多模态特性，充分利用手部信息以外的唇动、表情、头势等信息，研究基于多模态的多示例学习方法；针对视角变换和用户变换情况，采用半监督学习机制，利用大量无标注数据更新模型参数，来提高系统的可移植性。

基于流形学习、度量学习、低秩估计、字典学习、交叉验证等方法，解决手势手语识别中的多视角问题、自适应问题、快速识别问题、非特定人识别问题等；项目组成员共发表高水平论文14篇，申请发明专利7项，参加国际会议2次，参加国内会议2次。依托于项目的核心技术，项目组开发了钓鱼网站检测系统和场景图像文字检测识别系统，前者应用于工信部安全中心，后者应用于中央网信办。