数据中心内关联性流量的协同管理方法研究

To date, systems of massively distributed computing frameworks, such as MapReduce, Dryad, CIEL, Pregel, and Spark have been widely used in large-scale data centers. These systems typically follow a data flow computation paradigm, where massive data are transferred across successive processing stages. Data transfers, such as the common multicst, incast, and shuffle communication patterns, contribute most of the network traffic in MapReduce like working paradigms and thus have severe impacts on application performances in modern data centers...In this project, we target at managing and optimizing the network activity at the level of transfers to aggregate correlated data flows and thus directly lowering down the network traffic resulting from such data transfers. More precisely, we will address four key challenging issues as follows. Firstly, we will study the cooperative scheduling and in-network data aggregation problem of such transfers in representative server-centric and switch-centric data centers, and then propose approximate approaches to minimize the resulting network traffic of such data transfers. Secondly, we will form the uncertain data transfers by leveraging the endpoint flexibility. We then study the cooperative scheduling and in-network data aggregation problem of such data transfers, and minimize the resulting network traffic by joint optimization the allocation of computation and network resources, and the allocation of storage and network resources. Thirdly, we found that the cooperative scheduling and in-network data aggregation face the dynamic and fault-tolerant issues. Therefore, we propose efficient self-adaptive approaches for the cooperative scheduling and in-network data aggregation schemes to ensure their usability and correctness. At last, we study the scalable implementation problem of the cooperative scheduling and in-network data aggregation scheme in practice, and propose one fundamental approach and two bloom filter based scalable approaches.

数据中心网络得到了迅猛发展，网内东西向流量管理是最重要的基础问题之一。本项目从基本理论和实现方法两个角度研究关联性流量协同管理四个基础科学问题：研究关联性流量的协同传输和网内聚合问题，为incast、shuffle、multicast流量提出最小化传输代价的协同传输理论方法，在大幅降低网络传输开销的同时缩短网络应用的完成时间；研究不确定性关联流量的协同传输和网内聚合问题，提出不确定性multicast 和incast 类型流量的协同传输理论方法，通过网络、计算和存储资源的联合优化最大化关联性流量的网内聚合效果；首次发现关联性流量的协同传输和网内聚合机制面临动态性和容错性问题，提出网内聚合机制对动态关联性流量和网络故障的自适应机制，确保提出的相关理论方法的有效性和可行性；研究基于协同传输的网内聚合可扩展实现问题，确保数据中心对该网内聚合应用提供底层支撑，使其在理论和实际上具备可扩展能力。

数据中心网络得到了迅猛发展，网内东西向流量管理是最重要的基础问题之一。本项目从基本理论和实现方法两个角度研究关联性流量协同管理四个基础科学问题：研究关联性流量的协同传输和网内聚合问题，为incast、shuffle、multicast流量提出最小化传输代价的协同传输理论方法，在大幅降低网络传输开销的同时缩短网络应用的完成时间；研究不确定性关联流量的协同传输和网内聚合问题，提出不确定性multicast 和incast 类型流量的协同传输理论方法，通过网络、计算和存储资源的联合优化最大化关联性流量的网内聚合效果；首次发现关联性流量的协同传输和网内聚合机制面临动态性和容错性问题，提出网内聚合机制对动态关联性流量和网络故障的自适应机制，确保提出的相关理论方法的有效性和可行性；研究基于协同传输的网内聚合可扩展实现问题，确保数据中心对该网内聚合应用提供底层支撑，使其在理论和实际上具备可扩展能力。在本项目支持下，团队共发表和录用34篇论文，其中ACM/IEEE Transactions论文16篇，32篇被SCI检索或待SCI检索；获得授权国家发明专利4项，申请国家发明专利5项。2020年，项目负责人郭得科教授牵头获得了湖南省自然科学一等奖，题为“适应性网络系统的设计优化理论”；2020年，项目负责人郭得科教授因在网络拓扑结构和高可用路由方面的成就，被授予CCF-IEEE CS青年科学家；2021年，项目负责人郭得科教授牵头获得了中国电子学会自然科学二等奖，题为“适应性网络系统的可扩展拓扑结构和控制平面”。