基于时空域流形一致性学习的监控视频超分辨率研究

In recent years, the national security situation is becoming increasingly complex. While the monitoring objects were often far away from the camera in various criminal cases. Images from surveillance video are often too small to recognize by people that cannot meet the needs of criminal investigation. Taking full use of the spation-temporal multi-manifold constraints is is the important direction to break the bottleneck of super-resolution. However, multi-manifold constraints brings inconsistency of manifolds.Due to lacking of enough prior information, the quality of reconstructed HR image is not ideal to furthermore criminal investigation. In order to solve this problem, this study intends to promote traditional manifold-learning based SR algorithm from spatial image blocks to spatio-temporal video tensor, then the temporal correlation and spatial content relevance constraints are combined for SR reconstruction. First, when the spatio-temporal manifold are heterogeneous results to inconsistency of embedded manifolds. This study intends to build a novel sparse tensor analysis model for spatio-temporal manifold geometry maintaining which can improve the consistency of both HR and LR manifolds. Secondly, high-dimensional video tensor exacerbated the level of manifold inconsistency between HR and low–resolution(LR) images, HR map topology and group structure are used to constraint the representation dictionary learning which improves the consistency of representation of manifolds. Finally, multi-scale of spatio-temporal manifold are iterative approximated to match the HR manifold structure, which improves the subjective and objective quality of video SR reconstruction. Results from this research are expected to raise the objective and subjective quality of reconstruction image with 1 MOS points, and 0.8dB respectively. Our algorithms attempt to solve the major technology issues in the application of city's surveillance systems, and significantly enhance the practical performance of surveillance system.

近年来国家安全形势日趋复杂，夜间、雾霾等恶劣成像环境中各类大案要案目标对象视频质量差、分辨率低无法满足刑侦业务的辨识需求。综合利用监控视频的时域多帧和空域样本先验是突破现有超分辨率技术瓶颈的重要方向，然而，时空域多先验的高维度引起了高低分辨率流形的不一致，限制了重建图像质量的进一步提升。针对这一问题，本研究将流形学习超分辨率技术中的空间图像块推广到时空视频张量，提出融合时空域多先验的流形一致性超分辨率方法。首先，研究时空域稀疏平滑嵌入规律，引入时空域视频块稀疏张量分解模型，保持时空域的几何结构，提升流形嵌入结构一致性；其次，引入不受降质影响的高分辨率样本局部结构和时域群结构监督字典学习，提高流形表达的一致性；最后，提出时空流形结构的多尺度迭代嵌入算法，渐进更新重建模型的尺度及权重，提升视频重建质量。本研究预期将视频监控图像重建主客观质量分别提高0.8MOS分和1.2dB，提升监控视频效能。

超分辨率技术能够提高图像/视频的分辨率和质量，在遥感观测，安防监控，人脸识别等领域具有广泛的应用前景。在实际的复杂应用场景中，图像受到噪声、运动模糊、低照度条件等降质因素的影响，获得的低分辨率图像常常难以辨识，给实际基于图像内容的信息感知类应用造成了巨大的技术挑战。本项目深入研究了视频/图像超分辨率中高低分辨率流形一致性内在分布规律，从流形一致性学习的理论构建，数据驱动的先验知识获取、先验约束函数的求解、高低分辨率映射关系等关键问题展开了研究，形成了一套适用于实际场景条件的先验自适应超分辨率理论，同时在低分辨率视频/图像重建、低分辨率目标识别等领域取得了关键性技术突破，预期成果可以应用到军事、遥感卫星、安防监控等领域。具体的成果如下：.1.建立对噪声、干扰鲁棒的局部低秩先验表达模型，在超分辨率重建算法中引入基于局部低秩表达，实现对表达样本的多流形聚类，使得算法对输入的噪声等干扰信息具有良好的鲁棒性，相比对比算法提升了PSNR值0.58db，解决了传统方法对输入噪声、干扰敏感、先验表达偏移导致的重建精度下降问题。2.建立了多层先验表达的统一模型与方法，提出了基于先验深度表达的超分辨率算法，引入多层表达字典的学习和更新模式逐层提升表达系数的表达精度，相比对比算法提高了客观重建质量1.48db。3.发展了基于多尺度残差神经网络的超分辨率算法，提出了时间和空域上的多尺度重建网络，提升了视频和图像的超分辨率重建性能，有效的解决了现有深度学习方法对单一尺度细节信息重建能力的不足问题。4、研发了面向实际低分辨率输入图像的人脸识别方法，实现极低分辨率的人脸图像识别，在极低分辨率测试图像上提升了4.9%的识别率。发表TMM、TIP等权威期刊论文17篇和ICME、ICASSP等高水平会议论文17篇，其中ESI高被引论文1篇，出版专著1部，申请发明专利10项，授权发明专利4项，获得IFTC2018会议最佳论文奖。对于实际超分辨率算法提供了通用可行的解决方法。