基于超级计算机的大数据处理支撑平台研究

In recent years, the complex applications consisting of both high-performance computing and big data processing phases come to emerge gradually. The platform able to support both the two computing models is on demand to support applications of this kind. Generally, the distributed cluster used to support big data processing is unable to support high-performance computing applications efficiently. Therefore, this research aims at facilitating the supercomputer designed for high-performance computing to support big data processing, thus coupling the two types of applications on the supercomputer. There exist three challenges when running big data processing jobs on supercomputer: ①the large amounts of files produced by big data processing introduce so heavy workloads that are beyond the capability of the storage system in supercomputer; ②the hybrid job scheduling taking both high-performance computing and big data processing into account confronts with unimagined challenges; ③the big data processing framework must be redesigned to adapt to the architectural characteristics of supercomputer. By considering the three challenges mentioned above, this work focuses on the research of the big data processing-oriented globally-shared storage system, the hybrid job scheduling of high-performance computing and big data processing, and the big data processing framework based on supercomputer. The technologies involved in the research will help to build a supercomputer-based platform supporting big data processing, and ultimately couple high-performance computing and big data processing on the supercomputer.

当前，一些复杂应用同时包含高性能计算和大数据处理两种计算模式，这就要求将两种计算模式统一到一套平台上。然而目前普遍用于大数据处理的分布式集群并不能有效支持高性能计算，因此，本项目试图使面向高性能计算的超级计算机支持大数据处理，从而实现两种计算模式在超级计算机上的融合。在超级计算机上实现大数据处理存在以下三大挑战：①大数据处理应用产生的海量文件将对超级计算机的全局并行文件系统形成巨大的压力；②在融合平台上实现高性能计算作业和大数据处理作业的混合调度面临严峻挑战；③在超级计算机上实现便捷高效的大数据处理还有赖于应用框架的全面革新。针对以上三大挑战，本项目分别从面向大数据处理的全局共享存储系统、高性能计算与大数据处理混合调度系统、基于超级计算机的大数据处理框架三个方面展开研究，构建基于超级计算机的大数据处理支撑平台，最终实现高性能计算与大数据处理在超级计算机上的融合。

高性能计算与大数据处理的融合是超算应用发展的新趋势，为此需重构超算系统软件以满足新的需求。本项目围绕超算上的大规模存储、资源管理与作业调度、大数据处理框架展开研究，促进高性能计算与大数据处理在超算上的融合。具体地，项目研发了大规模可扩展并行文件系统Kapok，同时解决了海量小文件低延迟随机访问和共享大文件高带宽并发访问问题，能够同时支持高性能计算和大数据处理负载，保证了数据存储和管理的融合。突破了物理资源和虚拟资源混合管理技术，提出了Push模式和Pull模式相结合的异构作业调度方法，实现高性能计算作业和大数据处理作业的统一调度，保证了调度上的融合。面向科学大数据的高维度有拓扑特征研发了科学大数据处理框架，针对超算的全局共享存储和处理器的NUMA结构优化了Spark系统，显著提升超算上的大数据处理效率。所研发的系统在超算中心部署使用，支撑了多领域的大量应用。项目发表论文29篇，其中1篇论文被评为中国科协第五届优秀论文，申请专利15项，其中授权11项（含PCT专利5项），登记软件著作权7件。项目成果作为重要支撑材料获得中国电子学会科技进步一等奖和“奥林帕斯”先锋奖。