基于Hadoop的气象云存储与数据处理平台的研究

Establish meteorological clouds for meteorological industry is an efficient measure to implement data sharing, intelligent storage, backup and data processing of mass meteorological data. However, there are some problems at present, such as unstable of cloud storage structure, different methods of classifying. Both of them will impact the reliability of the cloud platform and management performance. This project aims at studying the management of meteorological cloud storage and data processing platform based on Hadoop. It contains three parts. The first part needs to find the processing method to handle single point of failure for the metadata that exists in the underlying architecture of the meteorological cloud storage platform, the solution of data nodes' intelligent management and the data's backup and storage solution blending with the mechanism of "disaster-tolerant at distance". The second part tries to find the way that how meteorological data can storage by categories and data backup by classification. The last part attempts to study the processing that how to deal with the loose coupling meteorological jobs of "general computing" by Task Scheduling Model which can be solved by Heuristic Algorithm. The project will finally improve the underlying architecture of Hadoop clusters in current meteorological cloud platform and also develop the meteorological data storage, thereby it will lay a solid theory and technique foundation for meteorological industry to establish "proprietary meteorological cloud" service.

为气象行业建立"气象云"是实现海量气象资料共享、智能存储、备份以及数据处理的高效手段。然而，当前云存储架构存在的各种不稳定因素以及气象资料的不同分类、分级方法将会影响云平台的可靠性和管理性能。本项目重点研究基于Hadoop的气象云存储管理与数据处理方法与技术。主要研究内容有：气象云存储平台底层架构中元数据节点单点故障处理方法，数据节点智能管理的解决方法以及融合"异地容灾"机制的数据备份存储方案；结合观测气象资料的种类以及使用级别，研究气象云平台上气象资料分类存储、数据分级备份方法；研究具有"松耦合"特性的"通用计算"类气象作业的任务调度处理模型以及启发式算法求解该类模型的过程。本项目最终将完善当前气象云平台Hadoop集群的底层架构，从而为气象行业建立"专有气象云"服务奠定坚实的理论和技术基础。

随着云计算技术在气象行业的应用与推广，云计算在气象行业的应用面临着诸多挑战，主要包括：（1）基于Hadoop的气象云存储平台中，气象元数据节点与气象数据节点瘫痪；（2）气象部门资料存储无法共享，存储备份调用效率低下；（3）由于资源需求量较大，“通用计算”类气象作业处理效率低下的问题。本研究课题主要针对上述挑战进行研究，主要研究工作包括：（1）研究了气象云存储平台底层可靠性架构及相关核心技术；（2）研究了气象资料分类存储、检索以及数据分级备份的解决方案；（3）研究了云计算环境下的气象作业任务调度处理模型和资源调度方法。本课题的研究有助于基于开源云平台Hadoop技术构建气象云行业平台以及备份云平台，并从理论与实践相结合的角度，通过多个气象实例分析与研究，从而实现气象云平台数据处理分析研究，进而可以为气象行业建立气象行业的云服务奠定坚实的理论和技术基础。