大数据流式计算的网络调度优化理论与算法研究

Big-data stream processing system enables very low-latency massive data processing, which has been broadly applied in the areas of Internet, Industry, Finance, etc. The main challenge of its design is to improve the stream task processing efficiency by smartly utilizing the system resources. Our project implements an operations research of distributed scheduling and resource allocation for big-data stream computing systems. (1) Based on the large-scale measurements of big-data stream processing systems, we observe the workload behavior, traffic variation and task patterns of data streams in real processing. (2) We analyze the system performance and its bottleneck based on the modeling of the system characteristics. (3) We propose to study the optimization models of resource management issues by means of mathematical theory and machine learning approaches. These issues include the task assignment optimization, dynamically distributed scheduling and load balance among heterogeneous clusters, etc. Based on the theoretical modeling, we design corresponding algorithms to improve the performance of distributed scheduling and resource allocation in the large-scaled networking system. Our study provides the theoretical foundation and practical suggestions for the system optimization of big-data stream processing, in order to achieve the high-throughput, low-latency and high-availability in operations.

大数据流式计算系统用于海量数据的实时处理，在互联网、工业和金融等领域有着广泛需求，如何有效利用系统资源提高实时任务的执行效率是其主要挑战。本项目研究大数据流式计算系统的网络调度优化理论及其算法。（1）通过对大数据流式计算系统的实际测量，观察流式数据处理的负载特征、流量特征以及连接特征等方面。（2）根据系统的基本特征建立模型，分析系统性能表现及其运行瓶颈。（3）结合数学理论和数据科学思想设计资源管理的优化模型，主要讨论大数据流式计算任务的分配优化、分布式动态调度以及异构资源的负载均衡等关键问题。基于理论模型的研究，设计高效地网络调度和资源分配算法，提高系统资源的利用率和系统整体的运行效率。为建立高吞吐、低延迟且持续可靠运行的大数据流式计算系统提供理论依据和实践指导。

本项目围绕大数据计算分析与算法应用展开。通过对实际大数据系统的测量观察，总结了数据传输模式及分布式应用特征。在此基础上，使用数学建模和数据建模工具对用户行为、网络流量、资源调度等主要问题展开分析和研究。并结合实际工程探索了大数据计算分析与深度学习模型的实践应用：（1）通过对海量视频内容分发网络的数据测量分析，发现用户使用移动网络的流量特征并建立模型改进传输性能。（2）结合测量数据分析探索效用模型，讨论了基于数据反馈的互助型内容传输体系，建立数学模型并在实际系统进行大规模应用试验，有效提升了带宽资源的利用率。（3）面向大数据问题的具体应用，探索了基于深度学习模型的互联网视频推荐和热度预测的分析方法，并结合实际工程场景获得检验。