Abstract:

［Purpose/Significance］New measurements for search engine evaluations should be investigated to deal with the shift that Web search engines like Google are evolving from the keyword-matching retrieval systems to the comprehensive platforms with functions of searching and natural language question answering(QA).
［Design/Methodology］Questions used in TREC QA Track were used to test the QA ability of Google both from quantitative and qualitative aspects.
［Findings/Conclusion］The results of qualitative evaluation show that Google has already acquired some QA ability in natural language. Based on content analysis of answers, the characteristics and the existing problems of search engines' QA ability have been revealed. Meanwhile, suggestions on how to promote the QA ability have been provided. ［Originality/Value］This paper has proposed a novel measurement to evaluate search engines from the perspective of natural language question answering ability.

Key words: Search engine, Natural language question answering, Information retrieval evaluation, Search engine evaluation, Questionanswering service