车联网中基于增强学习的边缘计算技术研究

Edge computing can provide low delay and high reliable service for vehicular networks. Short-term data stream congestions easily occur in vehicular networks for the time-varying topology. 5G has high data rates, however, the signal attenuates fast, and the coverage is small, and the cost of establishing 5G micro base stations is high. Unmanned aerial vehicles (UAVs) have the advantages of low price, easy deployment, flexible mobility, and most line-of-sight links; therefore, UAVs can complement 5G vehicular networks, acting as important mediums for connecting vehicular networks and edge computing servers. This project includes reinforcement learning based communication relay, traffic flow control, and edge computing resource management. Communication relay is that a UAV adjusts the 3D movement and communication resource allocation to maximize the total throughput; traffic flow control is that edge computing servers and clouds recognize vehicles based on UAVs' aerial images, and then compute control policies to adjust traffic lights to maximize the total traffic flows; edge computing resource management is that the UAVs allocates vehicles' tasks to edge computing servers or remote clouds to maximize the total utilities. The above problems are all Markov Decision Processes after analysis; however, their parameters are time-varying and hard to measure, therefore, deep reinforcement learning (DRL) is used. To make DRL be in accord with the particular features of the above problems, the frameworks of DRL are revised in terms of experience storage-sampling and optimistic estimate, large scale control, and reducing the risk of reward functions.

边缘计算可为车联网提供低时延、高可靠性服务。车联网由于其时变的拓扑结构易造成短时间数据流拥堵，5G虽然速率快但信号衰减快、覆盖范围小，建设5G微基站成本高。无人机具有低价格、易部署、灵活移动、大多为视距传输链路的优点，因此无人机可弥补5G车联网的不足，成为连接车联网和边缘计算服务器的重要媒介。本项目包括基于增强学习的中继通信、交通流控制和边缘计算资源管理。中继通信指无人机通过调节三维移动和通信资源分配以最大化总吞吐率；交通流控制指边缘计算服务器和云端根据无人机航拍图像进行车辆识别，然后计算出控制策略以调节交通灯最大化总交通流；边缘计算资源管理指无人机分配车辆的任务给边缘计算服务器或远程云端以最大化总收益。经分析上述问题均为马尔可夫决策过程，但其参数时变、难测量，所以采用深度增强学习。为了与上述问题的特有属性相契合，从经验存储抽样与乐观性估计、大规模控制、降低奖励函数风险三个角度改进了框架。

本项目将增强学习用于无人机辅助的车联网、大规模交通灯控制的研究，本人开发了增强学习库ElegantRL(github 2.6k星，链接：https://github.com/AI4Finance-Foundation/ElegantRL)及其子库ElegantRL_Transportation（链接：https://github.com/AI4Finance-Foundation/ElegantRL_Transportation）。通过本项目，证明了增强学习在车联网领域，能够达到优良的控制性能，在小规模上可以达到最优解，在大规模上明显好于传统方案。另外，ElegantRL具有高效率、高稳定性、云原生等特性，在多个领域得到了广泛的使用和好评，比如车联网、智能交通。