海量网络流量数据的跨时空域协同分析和性能优化研究

With the rapid development of software-defined networks (SDN) and data center networks (DCN), traffic measurement research which can assist the operation and management of network infrastructure is becoming increasingly popular in the past decade. Existing works on network traffic measurement mainly focus on deploying monitors in gateway routers to inspect the traversing traffic flows. They overlook the emerging need of monitoring both the crossing-gateway traffic and the intranet traffic, for the purpose of detecting network faults and malicious behaviors occurring in intranet. Therefore, our project aims to build a web of traffic monitors that cover both the gateway routers and the intranet devices, including routers and physical/virtual switches. With such densely deployed monitors, we are able to jointly analyze network traffic spanning multiple spatial and temporal domains, which in turn enables the detection of complex traffic anomalies. However, a major obstacle is the limited amount of resources available on each distributed traffic monitor. To allow the tradeoff between resource consumption and measurement quality, we will apply the theoretical tools of streaming data analysis and data sketching, in each of the three phases: collection of network traffic summary data, centralized analysis of multiple data summaries, and resource scheduling of traffic monitors. Specifically, in this proposal, we will study the following four research topics: (1) real-time and accurate analysis of large volumes of traffic flows with low resource consumption; (2) composite traffic monitor supporting multiple measurement functions, in order to extract multi-dimensional traffic features; (3) joint analysis of multiple data summaries spanning spatial and temporal domains; (4) on-demand scheduling of measurement functions and adaptation of flow sampling rates, to significantly reduce the resource consumption of programmable traffic monitors. Based on the above theoretical studies, we will build a prototype system in a real-world data center to verify our design of traffic monitor web and evaluate our joint analysis algorithms of network traffic crossing multiple spatial and temporal domains. Our project will accelerate the academic research on network fault diagnosis and anomaly detection based on real-time measurement data.

随着软件定义网、数据中心网、物联网的快速发展，基于网络测量的网络故障诊断和异常行为检测成为研究热点。现有工作主要关注在网络出口边界的高效流量测量和异常检测，忽视了企业内部网络自动化运维的需要，也无法检测侵入内网的复杂网络行为。为此，本项目以构建高效的分布式泛在流量探针为目标，基于流式数据分析技术，分别从流量数据采集、探针资源调度、多流量摘要的集中式分析入手，深入研究跨时空域的流量数据分析和分布式探针的资源优化：研究Gbps甚至Tbps级海量数据流的近实时、高精度、低开销测量；研究复合型多功能测量探针的高效执行，实现多维度流量特征的采集；研究跨时空域的数据流协同状态测量，支持复杂异常行为的检测；研究可编程流量探针功能的按需编排，实现低资源开销的流量分析。在此基础上基于实际计算中心环境开发支持跨时空域联合流量分析的探针网络的原型系统，为基于流量测量的网络异常检测提供理论和技术支撑。

骨干网和数据中心网会产生海量的流量数据。网络流量分析面临的一大挑战是流量数据规模可以达到Gbps甚至Tbps级别。这给流量数据的存储和传输都带来极大的困难。近年来，人们发现流量探针可以利用流式数据分析技术压缩海量报文流，实现低资源开销的流量分析。流式数据分析的目的是根据所需要的关键统计信号，快速生成海量流元素序列的摘要，抛弃掉细节的原始数据。同时将关键统计信号的损失控制在概率可接受范围内，保证最大限度的恢复原始信号特征。..我们提出了多种高精度、低开销、可实时查询的网络测量方法。.（a）单网络流的空间高效基数测量：基数估算问题是流式大数据处理算法的基石。我们将传统Hyperloglog算法的空间性能提升20%-50%，并解决了其整体估算域上存在偏置的问题。工作发表在A 类会议IEEE INFOCOM'17和投稿A类期刊ACM/IEEE TON 2020。被英国Axiom公司实现和用来分析时序数据的基数。据计算机黑客和创业公司新闻网站Hacker News，该算法成为最受欢迎的Hyperloglog算法改进。.（b）海量网络流的空间高效基数测量：该问题估计每个网络流的基数，为网络异常检测（端口扫描攻击、DDoS攻击、蠕虫感染等网络攻击）提供重要信息。我们的算法空间消耗比传统算法降低十倍，发表在网络测量领域顶级会议ACM SIGMETRICS'15和A 类期刊IEEE/ACM TON 2017。研究成果被内存数据库Redis和时序数据库InfluxDB采纳标准模块，用于检测top-k大基数超点。2021年的IWQoS论文进一步将海量网络流基数查询的访存开销降低到常数级，可实时检测超点。.（c）海量网络流的频数分布流矩的高效测量：网络流的频数常常用来检测DDoS攻击和快速网络扫描等高聚集的暴力型攻击行为。传统的被攻击目标检测是依靠采样Top-K的高频数流。对海量网络流的频数的整体分布缺乏感知。我们设计了复合测量Sketch算法，在检测Top-K大流的同时，测量频数分布的矩函数。论文发表在A 类会议IEEE INFOCOM’20和A类期刊IEEE/ACM TON 2023。..设计的算法实现在网络设备的数据转发面，比如P4可编程Intel Tofino交换机和DPDK软件交换机，获得了江苏省科技奖一等奖，以及获得Intel举办的P4编程比赛全国一等奖。