维吾尔、哈萨克、柯尔克孜文跨语言信息检索技术研究

Xinjiang region is rich of information provided in minority languages, which belong to same language system with languages being used in neighbor countries. With the rapid development of informatization of minority languages and popularization of Internet, increasing number websites have been emerged with high quantity of content in minority languages. Meanwhile, as a demand of the times, an urgent need for a proper search engine which could enables us obtain valuable information from these websites efficiently has been rose greatly, and been important problem needs to be solved. Although many domestic and oversea researchers have been working on Information Retrieval technologies, and, as result, many relevant algorithms have been proposed, but no work aimed at minority languages such as Uyghur, Kazakh and Kirgiz has been done systematically. We proposed a project which aims at framing language model, researching main components of Uyghur, Kazakh and Kirgiz Search Engine, and, for which, we need to apply several computer science fields including Probability and Statistic, Data Mining, Web Crawling and computational linguistic etc. Moreover, we systematically research Uyghur, Kazakh and Kirgiz stemming technology, information retrieval model, ranking model and generation of grid dictionary of these three languages to solve and apply key technologies in cross-cultural information diffusions by taking account of linguistic features.

新疆民族语言信息资源十分丰富，而且这些语言与周边国家语言同属一个语族，随着民族语言信息化和网络化迅速发展，境内外民族文网站层出不穷，网上信息急剧增长。随之而来的问题是由于缺乏好的信息检索系统，快速、准确、全面、方便地检索并获取有用的民族语言网络信息是时代的要求，是我国目前面临的一个急待解决的重要问题。目前国内外的众多研究者已对信息检索技术展开了深入研究，提出了很多算法，但还没针对维吾尔、哈萨克、柯尔克孜跨语言信息检索技术展开系统地研究工作。本项目拟以语言模型为框架，研究维吾尔、哈萨克、柯尔克孜文跨语言检索系统关键技术为目标，应用基于统计、数据挖掘、网络爬行、计算语言学等理论与技术，针对维吾尔、哈萨克、柯尔克孜等语言特点系统地研究维吾尔、哈萨克、柯尔克孜文词干析取、信息检索模型、检索结果排名模型以及维吾尔、哈萨克、柯尔克孜文关联词典的构建等，解决实现维吾尔、哈萨克、柯尔克孜文跨信息的关键技术

本课题将维、哈、柯跨语言搜索作为研究任务，围绕维、哈、柯多语言多文字自动识别及转换，多语种索引，查询扩展及搜索等方面开展了以下几个方面的探索新、创新性研究。关于维、哈、柯语种识别提出了一种基于统计专用字符和复合字符等特征的维、哈、柯文文种识别方法，从而达到了多语种批量采集、分类分析及处理目的。研究了一种基于Web Service的文本校对方法，并设计了一个根据用户记录来自我学习，自我优化的校对系统。关于搜索结果的分类，提出了一种基于N 元模型的文本分类方法，并在本搜索引擎上引用得到了较高的准确率。为了克服词干提取在索引及匹配上的消极影响，研究了基于字符N元模型的维吾尔文全文检索方法，验证了其可行性和有效性。关于查询扩充，提出了基于WordNet的自动生成维吾尔语的语义词典方法，以及基于语义词典的查询扩充方法，从而明显提高了搜索查全率。经过项目实施，培养了硕士生5人，构建了多个资源库，开发了多个算法，工具和平台,已发表学术论文6篇（EI收录1篇），投稿在审论文2篇。