文件流型大数据的分析模型构建与知识发现研究

There are a lot of data presented in the form of data file stream in the field of communication, Internet and monitoring. The analysis and mining for such data are faced with the following challenges: 1) Because of the high data redundancy and low value density, it is difficult to conduct the statistical analysis and mining on them directly; 2) Online aggregation analysis method is scarce, and the value of the data cannot be directly obtained; 3) In the scene of typical analysis mining, there are few incremental knowledge discovery frameworks and general algorithms. In view of the above problems, this project studies the key technology of file stream data oriented analysis model construction and knowledge discovery: 1) We provide the solution of data domain analysis and extraction, thematic dimension establishment and loading, multi-dimensional and multi-level analysis model construction; 2) We propose a multi-dimensional and multi-granularity online aggregation method, which obtains efficient online aggregation algorithm by exchanging user 's statistical requirements; 3) We explore the incremental knowledge discovery algorithm based on the online aggregation analysis, and give the knowledge discovery algorithm for typical file stream big data, combined with the relevant statistical methods, the technology of machine learning, data mining and deep learning. The achievements of this project can provide a general solution for data processing and analysis for file stream big data, which will be extended to the field of generalized big data research and further applied to multi-industry public management decision-making.

文件流型大数据广泛存在于通信、互联网和监控等领域，对其进行分析挖掘主要面临三个问题：1)数据冗余度高、价值密度低，难以直接统计分析与挖掘；2)在线聚集分析方法匮乏，无法直接获取数据中蕴含价值；3)典型分析挖掘场景下，缺乏增量知识发现框架和通用算法。针对上述问题，本项目研究文件流型大数据的分析模型构建与知识发现关键技术：1)给出数据域解析与抽取、主题维建立与装载、多维度多层次分析模型构建的解决方案；2)提出多维度多粒度的在线聚集分析方法，通过交互式获取用户统计需求，提出高效在线聚集算法；3)探索结合在线聚集分析的增量知识发现算法，应用相关统计方法、机器学习、数据挖掘和深度学习技术，给出典型文件流型大数据的知识发现算法。本项目研究成果可为文件流型大数据的数据处理与分析挖掘提供通用解决方案，最终将推广到广义大数据研究领域，并进一步应用于面向多行业的公共管理决策。

文件流型大数据广泛存在于通信、互联网和监控等领域，对其进行分析挖掘主要面临三个问题：1)数据冗余度高、价值密度低，难以直接统计分析与挖掘；2)在线聚集分析方法匮乏，无法直接获取数据中蕴含价值；3)典型分析挖掘场景下，缺乏增量知识发现框架和通用算法。针对上述问题，本项目研究文件流型大数据的分析模型构建与知识发现关键技术：1)给出数据域解析与抽取、主题维建立与装载、多维度多层次分析模型构建的解决方案；2)提出多维度多粒度的在线聚集分析方法，通过交互式获取用户统计需求，提出高效在线聚集算法；3)探索结合在线聚集分析的增量知识发现算法，应用相关统计方法、机器学习、数据挖掘和深度学习技术，给出典型文件流型大数据的知识发现算法。本项目研究成果可为文件流型大数据的数据处理与分析挖掘提供通用解决方案，最终将推广到广义大数据研究领域，并进一步应用于面向多行业的公共管理决策。