连续动作空间深度Actor-Critic算法研究

By integrating the advantages of the perception of deep learning and the decision making of reinforcement learning, deep reinforcement learning provides a new way for high-dimensional sequential decision-making problems. However, because the research time is so short, deep reinforcement learning is facing many difficulties, such as the sample decorrelation, the convergence guarantee, the computational cost, the learning efficiency, etc. Moreover, most of the proposed algorithms are designed for the sequential decision-making problems in discrete action spaces, and they are difficult to meet the practical engineering application. To overcome these difficulties, this project will investigate deep actor-critic algorithms in depth. Firstly, we will propose some novel random sampling mechanisms to remove the correlation among sequential samples, and rebuild the computational models of value function, advantage function and policy gradient. Secondly, we will analyze the asymptotic convergence factors of deep neural networks, and present the convergent conditions and the proof method about deep actor-critic algorithms. Thirdly, we will optimize and reconstruct the network architecture of deep actor-critic algorithms for reducing the computation cost. Lastly, we will design some adaptive vector-valued step-sizes for improving the learning efficiency of deep actor-critic algorithms. The research results of this project will not only enrich and deepen the theory of deep reinforcement learning, but also improve its engineering practicability.

通过将深度学习的感知能力和强化学习的决策能力有机结合，深度强化学习为求解高维序贯决策问题提供了新的思路。然而由于研究时间不长，当前深度强化学习在样本去相关性、收敛保证、计算开销和学习效率等方面还存在许多问题，且所提算法大多面向离散动作空间序贯决策问题，难以满足实际工程需要。为此，本项目主要围绕连续动作空间深度Actor-Critic算法展开研究。首先，建立可自动消解序贯采样相关性的随机取样新机制，重构值函数、优势函数和策略梯度计算模型。其次，对深度神经网络的渐近收敛影响因素进行分析，给出深度Actor-Critic算法的收敛条件和证明方法。再次，对深度Actor-Critic算法的网络模型结构进行优化改造，减小计算开销。最后，设计适于深度Actor-Critic算法的自适应向量值步长，提高学习效率。所获成果不但可以丰富和深化现有深度强化学习理论，而且可以有效提高其工程实用性。

与大多数深度强化学习算法相似，连续动作空间深度Actor-Critic（DAC）算法也容易遭受样本相关性、收敛性难保证、学习效率低等问题。为此，本项目从三个层面展开研究：首先，对DAC的底层算法进行研究，分别提出了适于五种常用神经网络的迷你批RLS优化算法集、面向大规模非受限优化的非单调缩放共轭梯度算法、0-1二次规划的神经动力方法、优化加权核范数模型方法、基于RLS优化的CNN裁剪算法。其中，我们的迷你批RLS优化算法可视作一类特殊的SGD算法，其综合运用了我们平均近似、等效梯度和经典的误差反向传播技术，其时间和空间复杂度和SGD同阶，但是却具有更好的收敛性能。其次，采用经验回放、迷你批学习方式或门限机制等消减样本相关性，提出了一系列基于RLS优化ESN、FNN、CNN和RNN的深度Q学习和Sarsa算法。最后，基于RLS-ESN和RLS-RNN, 我们提出了采用标准策略梯度的优势Actor-Critic（A2C）算法，其迷你批样本从经验回放集抽取。基于RLS-CNN，我们进一步提出了采用标准和自然策略的梯度A2C算法（即RLSSA2C和RLSNA2C），其迷你批样本由多个并行Workers与环境交互产生。实验结果表明，RLSSA2C和RLSNA2C不仅具有较标准深度A2C算法更好的学习效率，而且具有较当前主流深度A2C算法更高的计算效率。