面向云存储系统的硬盘故障预测方法研究

With the development of big data and cloud computing, cloud storage systems are quickly growing larger, which will experience disk failures more frequently. Predicting disk failures before they actually occur allows us to handle them in advance, which can significantly enhance the reliability and availability of systems. However, there are some drawbacks to the existing disk failure prediction methods. Therefore, for the current large-scale cloud storage systems, this project proposes new disk failure prediction methods which are more suitable for practical application. The main work are (1) new metrics for disk failure prediction models, which can directly reflect the impact of prediction on system performance; (2) high-efficient disk remaining life prediction models based on GBRT, and improving models based on the new metrics; (3) the reliability analysis methods to help building systems and prediction models, which can quickly and efficiently evaluate the statical and dynamic reliability of proactive cloud storage systems. These works are mingled to form a complete failure prediction, reliability analysis mechanism for cloud storage systems, which can promote proactive fault tolerance being applied in cloud storage system, and then significantly enhance the reliability and availability of systems.

作为大数据和云计算的基础支撑设施，云存储系统规模不断增大，引发系统中硬盘故障频发问题。硬盘故障预测通过对危险数据的提前预警和迁移，能够避免或减少故障带来的损失，保障系统的可靠性和可用性。然而，现有预测方法存在一些缺陷，未能真正应用于云存储中。因此，本项目拟面向大规模云存储系统，设计适合实际应用的硬盘故障预测方法，包括：（1）通过挖掘故障预测对云存储的影响，提出基于“数据保护”和“资源消耗”的硬盘故障预测模型评价指标，直接反映预测模型对系统性能的影响；（2）围绕新评价指标，研究GBRT硬盘剩余寿命预测模型和优化算法，量化指导预警处理；（3）采用组合分析、蒙特卡洛仿真和概率分析等方法，构造主动容错云存储系统的静态/动态可靠性评价模型，指导系统创建和预警处理。三者相互融合，弥补现有预测方法的缺陷，促进硬盘故障预测技术在云存储系统中的应用，从根本上提高系统可靠性和可用性，降低系统成本。

近年来，大规模云存储系统已成为主流IT企业的主要存储架构，在云存储系统中，硬盘故障已是常态，据估计，在一个Petabyte级别的文件系统中每天都会有硬盘故障发生，这给系统可靠性和可用性带来严峻挑战，也带来高昂的建设和管理成本。硬盘故障预测通过对危险数据的提前预警和迁移，实现系统的“主动容错”机制，可避免或减少硬盘故障造成的损失，能够从根本上提高云存储系统的可靠性和可用性。..本项目面向云存储系统，设计了适合实际应用的硬盘故障预测方法，具体研究内容包括：基于“数据保护”和“资源消耗”的新评价指标，直接有效地评价预测模型对系统可靠性和可用性的影响；提出创建基于GBRT和RNN的硬盘剩余寿命预测模型，并基于新的评价指标优化了模型训练算法，更细致准确地评价硬盘的健康状况，为后期预警处理的资源分配提供量化指导；提出主动容错存储系统的可靠性评价公式，定量评价故障预测对系统性能的影响，更科学、全面地指导预警迁移操作。..本项目在云存储系统硬盘故障主动预测方面取得了多项世界先进的研究成果，发表在IEEE TDSC、IEEE ACCESS、ICA3PP、航空学报等国内外期刊和会议上。项目培养了1名博士生、4名硕士生。同时后期获得两项产学研项目的支持，促进了相关领域研究的发展。项目提出一系列硬盘故障预测建模方法，在来自实际数据中心的数据集上，FDR达到95%以上，FAR小于0.01%，MR达到95%以上，MMR小于0.01%。