基于重影线索的单图像去反射关键技术研究

It often happens that one takes pictures through glass windows in real scenarios. The resulting images usually contain the reflection of the scenes behind the camera, leading to the degradation of image quality that affects the subsequent visual analysis. Since the reflection layer also contains the real scene content which is difficult to be distinguished from that of the background layer, removing the reflection from a single image is a very challenging problem. In recent years, there has been some progress in the study of deep-learning-based approaches for single-image reflection removal. Nevertheless, these methods have not effectively utilized the clues provided by the ghosting phenomenon which is very common in the images with reflection. Thus, there is still much room for performance improvement of the existing methods. This project aims to conduct in-depth and systematic research on the single-image reflection removal method based on the ghosting cues, which has the following focuses: (1) The construction of a double-reflection synthesis model which is more realistic than the existing ones; (2) Proposing a method for the accurate estimation of the parameters of ghosting phenomenon, which overcomes the uncertainty of parameter estimation of traditional methods; (3) The design of the regularization terms for the background layer as well as the reflection layer for effective discrimination between the two layers, with particular focus on remedying the ambiguity between the repetitive patterns of natural image and the ghosting patterns of reflection; (4) The construction of model-based deep network, which addresses the problem about how to effectively utilize the ghosting cues in deep networks. The development of the project has both theoretical values and practical values. It can provide new models, data and methods for image/video de-reflection research, as well as new ideas and new insights for the study on general image separation and deep learning. Moreover, the project can help to improve the performance of existing visual systems.

现实中人们常不可避免地透过玻璃拍照，所得照片常包含场景反射内容，导致图像质量下降，影响后续视觉分析。由于反射层包含实际场景内容，难以和背景层区分，从单幅图像中去除反射是一个极具挑战的问题。近年来，基于深度学习的单图像去反射研究有了一定进展，但这些方法并未有效利用图像常见重影现象所提供的线索，在性能上仍有很大的改进空间。本项目拟基于重影线索，对单图像去反射方法进行深入系统研究：（1）双重反射成像建模，构建更真实的模型;（2）重影参数准确估计，克服传统方法的测不准问题；（3）背景层和反射层正则化设计，实现两图层的有效区分，克服自然图像重复模式与反射重影混淆的问题；（4）基于模型的深度网络构建，解决如何在深度网络中有效利用重影线索的问题。项目的开展能为图像/视频去反射研究提供新模型、数据和方法，为广义图像分离及深度学习研究提供新思路，并有助于提升现有视觉系统性能，具有理论研究价值和应用价值。

图像去反射研究对改善图像质量和提高后续视觉任务性能有重要意义。本项目主要研究去反射等图像复原关键技术，围绕区分性正则化设计、最优化模型构建和不相关干扰因素去除分析等展开研究。本项目的重要成果如下：.1. 针对现有图像去反射方法未能有效利用重影线索，在性能上仍有很大改进空间的问题，提出了基于合成数据和真实数据混合监督训练的单图像去反射方法、基于反射区域导向的深度学习单图像去反射方法和基于形态滤波强化学习的图像去反射方法，提高了合成反射图像和真实反射图像的去反射效果。.2. 为了减少噪声、图层模糊和分辨率低等因素对去反射的不利影响，提出了基于多层次哈希的多源图像检索去噪方法、利用高层语义引导的图像去模糊及高分辨率重构方法和基于双注意力机制/前置白平衡的单幅图像去雨/雾方法。上述方法具有良好的去除噪声、模糊和低分辨率的效果，可以增强方法在复杂干扰因素下的鲁棒性，提高图像质量及后续检测识别任务的性能。.3. 为了减少不相关纹理对去反射等复原任务的干扰，提出了基于生成对抗网络和U型残差网络的复杂场景文本擦除和背景补全方法。场景文本擦除需要根据除文本位置外的已知背景纹理去预测空缺位置的背景纹理。这跟反射任务中反射特别严重的区域需要根据周围像素信息进行修补类似。该方法可以很好地迁移到去反射任务中，提高去反射等复原方法在不相关纹理等干扰因素下的鲁棒性，也有利于辨别背景层和反射层，为后续计算机视觉任务提供有用线索。.基于上述成果，项目开展期间总共发表了5篇高水平学术论文，其中JCR2区及以上论文4篇;申请了6项中国发明专利，其中授权2项;申请了1项软件著作权。项目的研究成果能为图像处理研究提供新模型、数据和方法，为广义图像分离及深度学习研究提供新思路，并有助于提升视觉系统性能，具有理论和应用研究价值。.