实时流数据处理系统中的弹性计算技术

Distributed stream processing system processes fast and continuous data stream in real time by leveraging the abundant resources in the cluster, and has been widely used in many applications, such as smart traffic, internet of things, and mobile Internet. In real applications, data arrival rate and key distribution usually fluctuate dramatically with time. Such dynamics may introduce unbalanced workload among the execution units and result in significant increase of processing time and reduce of throughput. We refer to the ability of handling workload dynamics immediately as elasticity. Existing systems achieve poor elastic, as they solve the workload imbalance by key space repartitioning of the input stream with high synchronization and state migration overhead. To solve this problem, this project aims to guarantee real-time data processing by exploring elastic computing, from perspectives of system architecture, performance model and implementation. We first study the method to decouple key space partitioning from resource allocation, to enable fast resource allocation updates. Then we study the performance model and estimate the workload and computational resource requirements of the executors under a given input data distribution. Finally, we study dynamic scheduling algorithm, which searches for a valid CPU allocation solution that minimizes migration costs, while constraining the computation locality. We believe this project will improve the system performance, in terms of throughput and processing latency, against a dynamic workload, and the theories and methods derived from this research work provides necessary guidance to the optimization on other distributed systems.

分布式流数据处理系统实时地处理快速、连续到达的数据序列，已被广泛用于物联网、移动互联网等领域。在实际应用中，数据到达的速率和数据的分布往往随时间剧烈波动，造成系统负载失衡、处理时间剧增、吞吐量骤降。现有系统应对负载变化时引入过高的的同步代价和计算状态迁移代价，导致调整速度滞后于负载的变化，丧失计算的实时性。本项目计划从系统架构、性能建模理论和算法设计等角度出发，研究流数据处理中的弹性计算技术，以保证负载波动时数据处理的实时性。首先研究弹性执行引擎，将计算资源的分配与数据流的划分解耦合，实现计算资源分配的快速更新；其次研究性能代价模型，准确计算执行单元的工作负载和资源需求；最后研究动态调度算法，搜索最优的资源分配更新方案，以维持计算的局部性、降低计算状态的迁移代价。本项目相关研究成果既为相关分布式系统的设计提供理论基础，又可直接应用到现有的系统中，具有较高的理论研究和实际应用价值。

分布式流数据处理系统实时地处理快速、连续到达的数据序列，已被广泛用于物联网、移动互联网等领域。在实际应用中，数据到达的速率和数据的分布往往随时间剧烈波动，造成系统负载失衡、处理时间剧增、吞吐量骤降。现有系统应对负载变化时引入过高的的同步代价和计算状态迁移代价，导致调整速度滞后于负载的变化，丧失计算的实时性。本项目计划从系统架构、性能建模理论和算法设计等角度出发，研究流数据处理中的弹性计算技术，以保证负载波动时数据处理的实时性。首先研究弹性执行引擎，将计算资源的分配与数据流的划分解耦合，实现计算资源分配的快速更新；其次研究性能代价模型，准确计算执行单元的工作负载和资源需求；最后研究动态调度算法，搜索最优的资源分配更新方案，以维持计算的局部性、降低计算状态的迁移代价。本项目相关研究成果既为相关分布式系统的设计提供理论基础，又可直接应用到现有的系统中，具有较高的理论研究和实际应用价值。