基于单目摄像头的空中手写汉字识别关键技术研究

The in-air handwriting is a new and more humanized human-computer interaction. Most of the existing methods of in-air handwriting human-computer interaction are based on large and expensive three-dimensional sensors such as Kinect or Leap Motion. This project intends to study the in-air handwritten Chinese character recognition key technologies based on low-cost monocular camera. Due to the lack of depth information in the data collected by monocular camera, and the influence of illumination change and occlusion, it is very difficult to collect writing trajectories. In this study, the applicant intends to use the sparse theory-based moving target detection algorithm as the basis and study the algorithm of detecting the position of the writing hand in video, so as to realize a robust method of forming in-air handwritten character. An in-air handwritten Chinese character is always finished in one stroke and has no pen-lift action. Traditional online Chinese character recognition algorithms can not recognize in-air handwritten Chinese characters very well. Therefore, the applicant intends to study a convolutional neural network classification algorithm suitable for in-air handwritten Chinese character recognition. In order to increase the number and diversity of training samples, a data augmentation technique based on generative adversarial networks is studied. The significance of our project is to explore the new technology of in-air handwritten character recognition technologies based on monocular camera, to promote the research and development of relevant subject, to provide a certain theoretical and practical evidence in order to further explore its application fields and expand its application scope.

空中手写是一种新的、更加人性化的人机交互方式。现有的空中手写人机交互系统大多是基于Kinect或Leap Motion等体积大且昂贵的三维传感器。本项目拟研究基于成本低廉的单目摄像头的空中手写汉字识别关键技术。由于单目摄像头采集的数据缺乏深度信息，另外光照变化、遮挡等影响使采集书写轨迹十分困难，在本研究中，申请人拟以基于稀疏理论的运动目标检测算法为基础，研究检测视频中书写手的位置的算法，实现鲁棒的采集书写轨迹的方法。空中手写文字都是一笔完成，没有起/落笔信息，传统在线汉字识别算法不能很好地识别空中手写汉字，因此申请人拟研究适合空中手写汉字识别的卷积神经网络分类算法。为了增加训练样本的数量和多样性，研究基于生成式对抗网络的样本生成算法。本项目的研究意义在于探索基于单目摄像头的空中手写汉字识别的新技术，也为推动相关学科的研究发展，进一步探索其应用领域，拓展其应用空间提供一定的理论和实践依据。

空中手写是一种新的、更加人性化的人机交互方式。现有的空中手写人机交互系统大多是基于Kinect或Leap Motion等三维传感器。此类传感器体积大，价格昂贵，不利于集成和大规模应用推广，本项目使用成本低廉、体积小的单目摄像头开发空中手写系统。. 由于单目摄像头获取的是没有深度信息的二维图像序列，本项目结合图像处理算法、稀疏编码相关理论和深度学习等研究了视频中运动目标检测与跟踪算法，提出了基于改进型YOLOv5的空中手写系统。基于该系统，本项目采集并构建了空中手写数据库IAHDB-AHUT2022，该数据库包括阿拉伯数字数据集、英文大小写字母数据集、汉字数据集、电视台名称数据集，总共58400个样本。为增加样本多样性，研究了基于生成式对抗网络的空中手写样本增强算法，有效提高了识别精度。为进一步提高空中手写汉字字符/文本识别精度，本项目提出了判别的端到端卷积神经网络空中手写汉字字符识别模型和端到端卷积循环网络空中手写文本识别模型，分别在公共数据集IAHCC-UCAS2016和IAHCT-UCAS2018取得了最优的识别性能，在传统联机手写上也取得了接近最优的识别精度，用了更小的存储消耗、提升了4倍的识别速度。. 本项目的研究成果具有广阔的应用前景，可以很方便得集成到手机、家用电器等，例如：可以代替遥控器通过隔空书写电视频道、空调温度等切换电视台、调节温度等。也可以用于军事隐蔽渗透过程中代替手势进行隔空书写交流，还可以用于课堂的趣味教学等等。疫情期间，银行签名等接触手写增加感染风险，利用空中手写可有效避免感染。也可以用于医院等高传染病区，避免接触感染。