支持多种计算与数据共享的编程框架研究

With the emergence of various domain-specific frameworks, it has become a trend for a powerful data processing platform to support multiple frameworks. To build such a unified platform, it is faced with a series of challenge. The framework manager can host a diverse of frameworks for resource sharing in a cluster. However, the framework lacks the built-in support for combining various computations and online data sharing. The project aims to solve these problems. We investigate a sequential mechanism and program structue for combining two widely-used computations (i.e.,DAG and BSP) in the same application program. We explore a distributed in-memory data sharing approach, allowing for access and mutate shared intermediate state via a common inferface. We present the system design and implementation for supporting the two mechansisms, and evaluate the solutions by extending our Transformer system. The study on this subject not only has great academic value on new programming frameworks, but also has directive significance for the software infrastructure development in data center.

随着大数据处理编程框架的不断丰富，在一个平台内支持多种类型的计算成为发展趋势，构建这样的大数据处理平台面临一系列的挑战。目前的框架管理器仅解决了多种框架共享集群资源的问题，但编程框架缺乏相应的机制，支持在一个应用程序内灵活地使用多种计算，以及在计算之间高效地共享中间数据。为解决上述问题，本项目研究一种串行机制及其程序结构，能在一个框架内支持常见的DAG和BSP计算；探讨一种基于内存数据集的共享机制，允许多种计算之间通过接口访问中间结果；提出一种系统架构与实现方法，能够同时支持这两种机制，并通过扩展已有的Transformer系统，对相关技术进行验证。本项目对研究新型编程框架有重要的学术价值；对数据中心大数据处理平台的研发有重要的指导意义。

近几年出现了各种各样的大数据处理框架。通常，从易编程、性能等角度看每个编程框架各有优缺点。因此，将多种编程模型组合在一起，可以互相取长补短，协同解决复杂的数据处理问题。但是，现有的编程框架只能通过松散的方式耦合在一起。针对大数据应用，我们提出了一种混合的编程框架-Transformer系统。相对于现有的技术，Transformer系统具有两个贡献：1）提供了一个统一的运行时环境，能实现多种编程框架；2）支持高效、透明的数据共享机制，能在一个程序内将不同的计算耦合在一起。在Amazon EC2机群上的测试表明，该系统能够支持大数据的混合编程。