During the practical labs in DITA session 04, I carried out an evaluation of information retrieval, focusing on how precise a set of results could be returned from queries using two search engines. Precision is measured by:
Relevant Documents retrieved / Total documents retrieved
Precision measures the quality of the results returned by a search engine. With searches for the term relevant is entirely subjective thus the number of relevant documents retrieved was a personal interpretation of what I deemed to be most relevant for each query.
Methodology
Twenty queries were used to evaluate the relevance of results returned using querying using both natural language searches and then combining natural language with Boolean logic operators and quotes (AND OR ""), first using www.google.co.uk and then www.bing.com.
The top 5 results were deemed to represent the total documents retrieved for the purpose of this evaluation.
The exercise was two fold, to try and classify the different types of information seeking needs or 'query types' and more importantly evaluate the produced by two search engines. IR is based upon the users needs and behavior.
The 3 main types of information need, as defined by Broder (2002) are:
Navigational queries: finding a home page of an organisation
Transactional queries: searcing for a service in order to make a transacton eg. www.ebay.com to purchase an item
Informational queries: to satisy a need for information for example how do I get rid of slugs in my garden?
The 20 queries used were:
1. Find the Website of Oxford University (Navigational)
2. Find the website of the 10th running of the International Society of Music Information Retrieval (Navigational)
3. Find the website of the of the organisation which represents Library Schools in the UK (Navigational)
4. Where can I buy bookcases in the UK? (Transactional)
5. Where can I buy sofa's in the London area? (Transactional)
6. Find sites where you can buy car insurance (Transactional)
7. Find sites where you can compare car insurance (Transactional)
8. Find the cheapest flight to Montevideo, departing next week, returning first week of January (Transactional)
9. Find the cheapest holiday to the Costa del Sol in Spain for the month of July (Transactional)
10 Who is the president of Uruguay? Can you find a biography of them? (Informational)
11. What are Ukuleles? (Informational)
12. Who was captain swing and what role did he play in early 19th Century English history? (Informational)
13. Try and find videos which provide some training on how to use search engines (as many search engines as possible). (Informational)
14. What are 'Jerusalem artichokes' and how do you cook them? Combining AND and the * wildcard could help to return relevance for answering 2 questions (Informational)
15. What are the origins of the Korean War (1950 to 1953)? (Informational)
16. How do hot air Balloons work? NB Boolean query also returned two videos on How hot ait balloons work (Informational)
17. What were the Putney Debates, and what was their impact? (Informational)
18. Who were the Levellers and what role did they play in the English Civil War?
19. Why did Hitler order the invasion of the Soviet Union in 1941 (Informational)
20. Find an image of a happy person. (Navigational)
Overall when the average precision for all search types were:
Using Natural Language: Google:70% Bing:75%
Adding Boolean operators: Google:71% Bing:78%
Analysing the precision of the three types of query, using Broder's Taxonony:
Google (no. of queries)
Navigational (4) 65.00% 46.67%
Transactional (6) 63.33% 70.00%
Informational (10) 76.00% 78.00%
Bing (no. of queries)
Navigational (4) 65.00% 70.00%
Transactional (6) 63.33% 63.33%
Informational (10) 86.00% 90.00%
Conclusions
The evaluation showed Bing returned a higher precision across all query types. In many cases the use of Boolean operators in queries increases the level of precision, with the exception of navigational type queries using Google. The way I used the Boolean operators was based on the success of the natural language results to try to improve precision.
One notable exception was query no. 13. Trying to find videos of search engine tutorials. The information need here was to find as many search engines as possible in the video, and when the natural language query returned videos of search engine strategies. Thus I thought it irrelevant and amended the query to:
Search engine tutorials NOT "search engine strategies" thus trying to omit results for search engine strategies and increase precision for the use of search engines. The query did not appear to omit search engine strategies, and indeed produced less precision. I am unsure why this to be the case.
When using SQL queries in structured data stored in a relational database the types of queries fall squarely in the 'Informational Need' category, but the information required will be discreet. In other words the ambiguity of using search engines in informational searches on the web, leads to a much deeper question, "how relevant is relevant?"
One thing I would like to mention about Bing, in that it is perhaps a much lesser used search engine, compared to the ubiquitous Google, is the use of pop ups that show"more of this page", thus allowing more judgment to be made on the relevance of that result. Google tends to provide a longer abstract, which is truncated and difficult to assess relevance of the result without clicking through to the page itself.
No comments:
Post a Comment