Stephen Robertson
About the author:
No description available of Stephen Robertson...
Publications by Stephen Robertson (bibliography)
» 2009 «
Robertson, Stephen, Vojnovic, Milan and Weber, Ingmar (2009): Rethinking the ESP game. In: Proceedings of ACM CHI 2009 Conference on Human Factors in Computing Systems 2009. pp. 3937-3942. Available online
“The ESP Game was designed to harvest human intelligence to assign labels to images -- a task which is still difficult for even the most advanced systems in image processing. However, the ESP Game as it is currently implemented encourages players to assign "obvious" labels, which can be easily predicted given previously assigned labels. We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one "off-limits" term assigned to them. We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.”
Copyrights may apply
» 2008 «
Cao, Guihong, Nie, Jian-Yun, Gao, Jianfeng and Robertson, Stephen (2008): Selecting good expansion terms for pseudo-relevance feedback. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2008. pp. 243-250. Available online
“Pseudo-relevance feedback assumes that most frequent terms in the pseudo-feedback documents are useful for the retrieval. In this study, we re-examine this assumption and show that it does not hold in reality -- many expansion terms identified in traditional approaches are indeed unrelated to the query and harmful to the retrieval. We also show that good expansion terms cannot be distinguished from bad ones merely on their distributions in the feedback documents and in the whole collection. We then propose to integrate a term classification process to predict the usefulness of expansion terms. Multiple additional features can be integrated in this process. Our experiments on three TREC collections show that retrieval effectiveness can be much improved when term classification is used. In addition, we also demonstrate that good terms should be identified directly according to their possible impact on the retrieval effectiveness, i.e. using supervised learning, instead of unsupervised learning.”
Copyrights may apply
Yilmaz, Emine, Aslam, Javed A. and Robertson, Stephen (2008): A new rank correlation coefficient for information retrieval. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2008. pp. 587-594. Available online
“In the field of information retrieval, one is often faced with the problem of computing the correlation between two ranked lists. The most commonly used statistic that quantifies this correlation is Kendall's τ. Often times, in the information retrieval community, discrepancies among those items having high rankings are more important than those among items having low rankings. The Kendall's τ statistic, however, does not make such distinctions and equally penalizes errors both at high and low rankings. In this paper, we propose a new rank correlation coefficient, AP correlation (?ap), that is based on average precision and has a probabilistic interpretation. We show that the proposed statistic gives more weight to the errors at high rankings and has nice mathematical properties which make it easy to interpret. We further validate the applicability of the statistic using experimental data.”
Copyrights may apply
Robertson, Stephen (2008): A new interpretation of average precision. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2008. pp. 689-690. Available online
“We consider the question of whether Average Precision, as a measure of retrieval effectiveness, can be regarded as deriving from a model of user searching behaviour. It turns out that indeed it can be so regarded, under a very simple stochastic model of user behaviour.”
Copyrights may apply
Ritchie, Anna, Robertson, Stephen and Teufel, Simone (2008): Comparing citation contexts for information retrieval. In: Shanahan, James G., Amer-Yahia, Sihem, Manolescu, Ioana, Zhang, Yi, Evans, David A., Kolcz, Aleksander, Choi, Key-Sun and Chowdhury, Abdur (eds.) Proceedings of the 17th ACM Conference on Information and Knowledge Management - CIKM 2008 October 26-30, 2008, Napa Valley, California, USA. pp. 213-222. Available online
Robertson, Stephen and Tait, John (2008): Karen Spärck Jones. In JASIST - Journal of the American Society for Information Science and Technology, 59 (5) pp. 852-854
» 2007 «
Mizzaro, Stefano and Robertson, Stephen (2007): Hits hits TREC: exploring IR evaluation results with network analysis. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2007. pp. 479-486. Available online
“We propose a novel method of analysing data gathered from TREC or similar information retrieval evaluation experiments. We define two normalized versions of average precision, that we use to construct a weighted bipartite graph of TREC systems and topics. We analyze the meaning of well known -- and somewhat generalized -- indicators from social network analysis on the Systems-Topics graph. We apply this method to an analysis of TREC 8 data; among the results, we find that authority measures systems performance, that hubness of topics reveals that some topics are better than others at distinguishing more or less effective systems, that with current measures a system that wants to be effective in TREC needs to be effective on easy topics, and that by using different effectiveness measures this is no longer the case.”
Copyrights may apply
» 2006 «
Taylor, Michael J., Zaragoza, Hugo, Craswell, Nick, Robertson, Stephen and Burges, Chris (2006): Optimisation methods for ranking functions with multiple parameters. In: Yu, Philip S., Tsotras, Vassilis J., Fox, Edward A. and Liu, Bing (eds.) Proceedings of the 2006 ACM CIKM International Conference on Information and Knowledge Management November 6-11, 2006, Arlington, Virginia, USA. pp. 585-593. Available online
Robertson, Stephen (2006): On GMAP: and other transformations. In: Yu, Philip S., Tsotras, Vassilis J., Fox, Edward A. and Liu, Bing (eds.) Proceedings of the 2006 ACM CIKM International Conference on Information and Knowledge Management November 6-11, 2006, Arlington, Virginia, USA. pp. 78-83. Available online
» 2005 «
Craswell, Nick, Robertson, Stephen, Zaragoza, Hugo and Taylor, Michael (2005): Relevance weighting for query independent evidence. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2005. pp. 416-423. Available online
“A query independent feature, relating perhaps to document content, linkage or usage, can be transformed into a static, per-document relevance weight for use in ranking. The challenge is to find a good function to transform feature values into relevance scores. This paper presents FLOE, a simple density analysis method for modelling the shape of the transformation required, based on training data and without assuming independence between feature and baseline. For a new query independent feature, it addresses the questions: is it required for ranking, what sort of transformation is appropriate and, after adding it, how successful was the chosen transformation? Based on this we apply sigmoid transformations to PageRank, indegree, URL Length and ClickDistance, tested in combination with a BM25 baseline.”
Copyrights may apply
» 2004 «
Hiemstra, Djoerd, Robertson, Stephen and Zaragoza, Hugo (2004): Parsimonious language models for information retrieval. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2004. pp. 178-185. Available online
“We systematically investigate a new approach to estimating the parameters of language models for information retrieval, called parsimonious language models. Parsimonious language models explicitly address the relation between levels of language models that are typically used for smoothing. As such, they need fewer (non-zero) parameters to describe the data. We apply parsimonious models at three stages of the retrieval process: 1) at indexing time; 2) at search time; 3) at feedback time. Experimental results show that we are able to build models that are significantly smaller than standard models, but that still perform at least as well as the standard approaches.”
Copyrights may apply
Upstill, Trystan and Robertson, Stephen (2004): Exploiting hyperlink recommendation evidence in navigational web search. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2004. pp. 576-577. Available online
» 2003 «
Soboroff, Ian and Robertson, Stephen (2003): Building a filtering test collection for TREC 2002. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2003. pp. 243-250. Available online
“Test collections for the filtering track in TREC have typically used either past sets of relevance judgments, or categorized collections such as Reuters Corpus Volume 1 or OHSUMED, because filtering systems need relevance judgments during the experiment for training and adaptation. For TREC 2002, we constructed an entirely new set of search topics for the Reuters Corpus for measuring filtering systems. Our method for building the topics involved multiple iterations of feedback from assessors, and fusion of results from multiple search systems using different search algorithms. We also developed a second set of "inexpensive" topics based on categories in the document collection. We found that the initial judgments made for the experiment were sufficient; subsequent pooled judging changed system rankings very little. We also found that systems performed very differently on the category topics than on the assessor-built topics.”
Copyrights may apply
» 2001 «
Craswell, Nick, Hawking, David and Robertson, Stephen (2001): Effective site finding using link anchor information. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2001. pp. 250-257. Available online
“Link-based ranking methods have been described in the literature and applied in commercial Web search engines. However, according to recent TREC experiments, they are no better than traditional content-based methods. We conduct a different type of experiment, in which the task is to find the main entry point of a specific Web site. In our experiments, ranking based on link anchor text is twice as effective as ranking based on document content, even though both methods used the same BM25 formula. We obtained these results using two sets of 100 queries on a 18.5 million document set and another set of 100 on a 0.4 million document set. This site finding effectiveness begins to explain why many search engines have adopted link methods. It also opens a rich new area for effectiveness improvement, where traditional methods fail.”
Copyrights may apply
» 2000 «
Robertson, Stephen (2000): On Theoretical Argument in Information Retrieval. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2000. p. 1. Available online
» 1995 «
Taghva, Kazem, Fox, Edward, Robertson, Stephen, Belkin, Nicholas J., Lewis, David and Harman, Donna (1995): Education for IR. In: Proceedings of the Eighteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1995. p. 358. Available online
“The SIGIR Education Committee has recently been formed based on the model of the SIGCHI committee which completed its final report in 1992. The committee is charged with developing curriculum recommendations for IR-related education to serve the computer science, library science, and information science communities. It will be soliciting input from information retrieval educators and the consumers of IR education (students and employers), with a view of determining the current status of IR education, the marketplace, and future direction. The committee is also interested in: * clearinghouses for IR courseware and training materials * electronic as well as traditional courses * demonstrations for online access to state of the art systems * other innovative efforts The purpose of this panel is to report briefly on the activities of the Education Committee and to stimulate discussion on the state of information retrieval education. The panel will consist of three IR educators from different communities (computer science, library science, information science) who will give brief (about 10 minute) presentations on their view of the purpose and content of IR curricula; a representative from the government sector will report on the role of IR in government agencies; and a participant from the industrial sector will consider the role of IR education in industry.”
Copyrights may apply
SHOW THIS LIST ON YOUR HOMEPAGE
What do YOU think?
Give us your opinion! Do you have any comments/additions
that you would like other visitors to see?
You say:
Mar 19th, 2010
Changes to this page (author)
18 Feb 2010: Enabled abstracts to be shown on Stephen Robertson's author page.01 Jun 2009: Author was edited 29 May 2009: Author was edited
29 May 2009: Author was edited
29 May 2009: Author was edited
09 May 2009: Author was edited
08 Apr 2009: Author was edited
08 Apr 2009: Author was edited
08 Apr 2009: Author was edited
12 May 2008: Author was edited
24 Jun 2007: Author was edited
24 Jun 2007: Author was edited
24 Jun 2007: Author was edited
24 Jun 2007: Author was edited
24 Jun 2007: Author was edited
24 Jun 2007: Author was edited
24 Jun 2007: Author was added to the bibliography
