基于多智能体深度自适应动态规划的优化控制方法与应用

As the expansion of modern control systems, the optimal control problem of the multi-agent system is becoming a new research hotspot, which puts forward new requirements for the design of the control system. For recent years, deep reinforcement learning/adaptive dynamic programming have made remarkable contribution in the field of optimal control for the individual agent and zero-sum game environment. As a typical method to deal with multi-agent systems in the simple environment, multi-agent reinforcement learning can be combined with deep adaptive dynamic programming effectively, which can provide solutions for the optimal control of multi-agent systems in complex game environments. Therefore, out project aims to study the key issues of multi-agent deep adaptive dynamic programming method. In theory and methodology, the experience replay will be improved to adapt the non-static environment of multi-agent systems. With the adaptive learning ability of multi-agent deep adaptive dynamic programming, the optimal control method for the nonzero-sum game with cooperation and competition and the cooperation game considering communication behaviors will be investigated, and theoretical analysis of system stability will be given. In experiment and application, mobile robot will be the main research objective of the proposed method. The simulation and real scenarios for the cooperative and operational task and car following task will be established. Then, validation experiments and comparative analysis for the proposed multi-agent deep adaptive dynamic programming will be given, which will lay the foundation of intelligent robot application.

随着现代控制系统规模的不断扩大，多个智能体间的优化控制问题逐渐成为新的研究热点，这也对控制系统的设计提出了新的要求。近年来深度强化学习/自适应动态规划在求解复杂未知环境的单个体优化控制和零和博弈问题方面做出了突出贡献。多智能体强化学习作为处理简单环境下博弈优化问题的典型方法，与深度自适应动态规划相结合，将为复杂博弈环境下的多智能体优化控制问题提供新的研究思路。本项目拟对多智能体深度自适应动态规划的关键问题进行研究，在理论方法方面，改进经验回放技术以适应多智能体非静态环境，借助多智能体深度自适应动态规划的自学习能力实现合作与竞争博弈下多智能体的协同对抗及合作博弈下考虑学习通信的协同优化方法，并对系统稳定性进行理论分析。在应用方面，以移动机器人为主要研究目标，建立多台移动机器人的协同作战与跟随行驶仿真及实际场景，对提出的多智能体深度自适应动态规划进行实验验证和对比分析，奠定智能机器人应用基础。

近年来深度强化学习/自适应动态规划领域快速发展，针对单智能体和多智能体环境均做出了突出贡献。本项目针对多智能体深度自适应动态规划的关键问题进行研究，首先在理论方法方面，针对多智能体斯塔克伯格博弈提出一种新的分层优化控制框架，保证闭环系统一致最终有界稳定的前提下该多智能体深度自适应动态规划算法可以近似获得斯塔克伯格博弈均衡。其次，针对合作与竞争博弈，提出了考虑内部奖赏的多智能体深度自适应动态规划算法QMIX-INTRI，在2V2的多机器人博弈对抗任务中对提出的算法进行实验验证和对比分析，相关工作获得国际会议2020 ICRA RoboMaster AI Challenge一等奖。此外，针对于机器人未知环境的自主探索，提出基于深度强化学习的探索策略，相关工作发布于国际顶级期刊IEEE TNNLS。进一步地，我们将相关方法扩展至自动驾驶的超车换道任务中，提出基于深度强化学习与规则约束结合、与虚拟样本结合的超车换道算法，发表多篇会议论文，并取得了国际会议2020 DAI SMARTS Competition竞赛第一名。总体而言，基于本项目参与人员共计发表论文14篇，包括6篇SCI论文和8篇EI论文，授权发明专利2项，获得了6项国内外比赛的奖项，培养研究生5名，顺利完成了项目目标和指标。