基于概念背景图的网络爬虫语义协作与竞争策略研究

In focused cralwing system, multi-crawlers crawl parallelly Web and download Web pages. It is one of hotspot research of search engine how the different focused crawlers avoid to visit the same URLs and they download efficiently Web pages related to the search topic. In order to accomplish rapidly the crawling tasks of the system for the specific topic, and embody fully every Web crawler's ability, we consider that these history visiting Web pages (URLs) of every focused crawler reflect their backgroup knowledge. On basis of cralwing independently, collaborating togather and competing with each other for Web crawlers of the system, we propose the novel understanding, cooperating and competing strategy of concept context graph by analyzing these Web page's content, extracting semantic features- - concepts of these Web pages in history collects of every Web crawlers as their backgroup knowledge and studing the semantic relationships of their backgroup knowledge. Our mainly researches are listed as follows:.1).Constructing the mathematical model of backgrounp knowledge of every Web crawler based on hierarchy concept context graph, according to the semantic characteristics- - concepts of Web pages and their semantic relationships among the concepts..2).Studying the understanding method and model among Web crawlers based on hierarchy concept context graph..3).Studying and implementing the cooprtating, competing model among Web crawlers of the same group managing by a F-Agent..4).Studying and implementing the cooprtating, competing model among Web crawlers of the diferent group managing by F-Agents.. By researching, we expect to acquire a series of new ideas, new methods, new technologies and new systems with understanding, cooperating and completing function among muti-agent Web crawlers for information retrieval. In search engine, this research project has important significances in theories and broad prospects in applications.

在多个Web主题爬虫并行爬行中，如何避免重复访问网页和高效地获取与主题相关网页，成为搜索引擎主题爬行的热点研究之一。为了完成系统爬行任务和充分发挥每个爬虫自身能力，本申请项目立足于每个爬虫相对独立爬行、共同协作、彼此竞争的思想，将爬虫的历史爬行网页作为背景知识，分析这些网页文本内容，提取网页的概念和概念间的语义关系,探讨不同爬行虫背景知识之间的语义相似性，提出基于分层概念背景图的爬虫之间理解方法、协作和竞争策略。重点研究四个方面的内容:1)主题爬虫背景知识的分层概念背景图的表示模型。2)基于分层概念背景图的爬虫语义理解方法。3)在语义理解模型下同组多个网络爬虫之间协作与竞争机制及实现。4)在语义理解模型下异组多个爬虫之间协作与竞争机制及实现。通过研究预期获得一套多网络爬虫相互理解、协作、竞争的信息获取的新思想、新方法、新技术、新系统。因此本项目研究具有着重要的理论意义和广阔的应用前景。

在多个Web主题爬虫并行爬行中，如何避免重复访问网页和高效地获取与主题相关网页，成为搜索引擎主题爬行的热点研究之一。为了完成系统爬行任务和充分发挥每个爬虫自身能力，本申请项目立足于每个爬虫相对独立爬行、共同协作、彼此竞争的思想，将爬虫的历史爬行网页作为背景知识，分析这些网页文本内容，提取网页的概念和概念间的语义关系,探讨不同爬行虫背景知识之间的语义相似性，提出基于分层概念背景图的爬虫之间理解方法、协作和竞争策略。重点研究四个方面的内容:1)主题爬虫背景知识的分层概念背景图的表示模型。2)基于分层概念背景图的爬虫语义理解方法。3)在语义理解模型下同组多个网络爬虫之间协作与竞争机制及实现。4)在语义理解模型下异组多个爬虫之间协作与竞争机制及实现。主要结果体现在：. 1)提出了基于用户兴趣Agent Crawlers初始种子集选择的方法.我们充分利用用户兴趣本体表达爬虫背景知识和用户感兴趣的区域作为初始种子集，实验结果表明该方法能有效的提高网页获取的进度。提出了最优概念背景图的概念,在在最优化概念背景图的基础上，提出了对概念背景图进行动态更新方法。. 2)提出了一种基于多Agent网络爬虫的理解度的衡量方法。这样可以从某种程度上消除自然语言的模糊性，提高理解的准确性。设计实现了基于多Agent系统的主题爬虫模型，使用改进的合同网协议模拟多Agent间的协调。实验结果表明基于多Agent系统的主题爬虫模型检索的网页相关性、相对回报率都有一定的提高。. 3)提出了适合Agent Crawlers合同网协议和拍卖协商协议。实验对比证明Agent Crawlers之间采用合适的协商方式可以显著地提高系统的效率。. 4)提出了多Agent的多对一、多对多关系组内组间竞争模型，该模型基于市场匹配中构造一组清仓价格的思想，让参与竞争的C-Agent与需要完成协作的资源形成完美匹配。实验表明查准率和查全率有一定幅度增长。