面向沉浸式感知的表面光场表示及传输

Virtual reality (VR) techniques aim at developing a brand new way of human-computer interaction and communication, and placing human in a virtual vision environment that is close enough to the reality, thus providing an immersive perceptual experience. This technology can be applied in various scenarios in human production and living applications including social communication, healthcare, education etc. It has attracted significant attention from academia and industry, thus accelerating the international standardization progress. However, existing VR techniques have not met the needs of human immersive perception. The essential issue is the fact that it is impossible to capture the whole perceptual spatial information and the lacks of efficient light field representation method, and the inevitable perceptual spatial information loss caused by the inefficient compression and transmission under the constrained bandwidth resources. This project targets at the core issue of light field representation, and will study the high-efficiency representation model based on the surface light field. This model supports rendering the scene from arbitrary viewpoints, and enables virtual reality the ability of describing immersive light field with high degree of freedom. Based on the model, this project will further investigate efficient, scalable and adaptive compression methods, unifying an integrated system framework for highly immersive light field representation and compression. These key issues will be studied theoretically and technically, such that the research results are expected to lay the foundation for the VR related applications and the standardization of light field, point cloud compression and transmission.

虚拟现实技术致力于发展全新的人机交互与通信方式，将人置身于接近真实的虚拟视觉环境中，给人以沉浸式的主观感知体验。该技术可以应用在生产生活的诸多领域包括社交、医疗、教育等等，得到了学术界和工业界的高度关注，从而促进了相关的国际国内标准制定。然而，现有的虚拟现实技术并没有满足用户对于沉浸感的要求。其本质的原因是无法获取完整的视觉空间信息，缺乏有效的光场表示方法，以及在带宽资源有限条件下低效率压缩、传输过程中不可避免的造成视觉空间信息的损失。本项目将从光场表示的核心出发，建立基于表面光场的高效表示模型。该模型可以从任意角度重构场景，从而赋予虚拟现实高自由度、沉浸式的描述能力，并针对该模型设计高效的、可伸缩的、自适应的编码方法，形成真正面向沉浸式感知体验的光场表示、压缩，为虚拟现实等相关应用的发展提供技术支持，也为光场、点云压缩、传输等国际国内相关标准化工作奠定基础。

本项目围绕表面光场数据，提出了一种高效的基于B样条小波基函数的表面光场紧凑表示方法，该方法可以使得表示系数稀疏、平滑且有利于压缩；进一步，提出了一种基于点云压缩框架的表面光场高效压缩方法，该方法具有空间和时域可伸缩等特性，相比于基于多视图像的方法，显著提升了率失真编码性能；此外，项目提出一种基于深度学习的光场图像压缩方法，该方法是一种基于生成对抗网络GAN的四维光场重建网络（Light Field Generative Adversarial Network, LF-GAN），能够高质量高效的根据已采样的视点对未采样视点进行恢复和重建。