Publication statistics

Pub. period:1986-2012
Pub. count:45
Number of co-authors:73


Number of publications with 3 favourite co-authors:

Aravindan Veerasamy:
Donna Harman:
David Lewis:



Productive colleagues

Nicholas J. Belkin's 3 most productive colleagues in number of publications:

W. Bruce Croft:124
Gary Marchionini:74
Susan Dumais:74

Upcoming Courses

go to course
Psychology of Interaction Design: The Ultimate Guide
Starts tomorrow LAST CALL!
go to course
User-Centred Design - Module 3
90% booked. Starts in 5 days

Featured chapter

Marc Hassenzahl explains the fascinating concept of User Experience and Experience Design. Commentaries by Don Norman, Eric Reiss, Mark Blythe, and Whitney Hess

User Experience and Experience Design !


Our Latest Books

The Social Design of Technical Systems: Building technologies for communities. 2nd Edition
by Brian Whitworth and Adnan Ahmad
start reading
Gamification at Work: Designing Engaging Business Software
by Janaki Mythily Kumar and Mario Herger
start reading
The Social Design of Technical Systems: Building technologies for communities
by Brian Whitworth and Adnan Ahmad
start reading
The Encyclopedia of Human-Computer Interaction, 2nd Ed.
by Mads Soegaard and Rikke Friis Dam
start reading

Nicholas J. Belkin

Has also published under the name of:
"Nichloas J. Belkin", "N. Belkin", "N. J. Belkin", and "N. J. Belkin"

Personal Homepage:


Publications by Nicholas J. Belkin (bibliography)

 what's this?
Edit | Del

Liu, Chang, Belkin, Nicholas J. and Cole, Michael J. (2012): Personalization of search results using interaction behaviors in search sessions. In: Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2012. pp. 205-214.

Personalization of search results offers the potential for significant improvement in information retrieval performance. User interactions with the system and documents during information-seeking sessions provide a wealth of information about user preferences and their task goals. In this paper, we propose methods for analyzing and modeling user search behavior in search sessions to predict document usefulness and then using information to personalize search results. We generate prediction models of document usefulness from behavior data collected in a controlled lab experiment with 32 participants, each completing uncontrolled searching for 4 tasks in the Web. The generated models are then tested with another data set of user search sessions in radically different search tasks and constrains. The documents predicted useful and not useful by the models are used to modify the queries in each search session using a standard relevance feedback technique. The results show that application of the models led to consistently improved performance over a baseline that did not take account of user interaction information. These findings have implications for designing systems for personalized search and improving user search experience.

© All rights reserved Liu et al. and/or ACM Press

Edit | Del

Liu, Jingjing and Belkin, Nicholas J. (2011): Search task difficulty: the expected vs. the reflected. In: Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2011. pp. 1183-1184.

We report findings on how the user's perception of task difficulty changes before and after searching for information to solve tasks. We found that while in one type of task, the dependent task, this did not change, in another, the parallel task, it did. The findings have implications on designing systems that can provide assistance to users with their search and task solving strategies.

© All rights reserved Liu and Belkin and/or ACM Press

Edit | Del

Cole, Michael J., Zhang, Xiangmin, Liu, Chang, Belkin, Nicholas J. and Gwizdka, Jacek (2011): Knowledge effects on document selection in search results pages. In: Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2011. pp. 1219-1220.

Click through events in search results pages (SERPs) are not reliable implicit indicators of document relevance. A user's task and domain knowledge are key factors in recognition and link selection and the most useful SERP document links may be those that best match the user's domain knowledge. User study participants rated their knowledge of genomics MeSH terms before conducting 2004 TREC Genomics Track tasks. Each participant's document knowledge was represented by their knowledge of the indexing MeSH terms. Results show high, intermediate, and low domain knowledge groups had similar document selection SERP rank distributions. SERP link selection distribution varied when participant knowledge of the available documents was analyzed. High domain knowledge participants usually selected a document with the highest personal knowledge rating. Low domain knowledge participants were reasonably successful at selecting available documents of which they had the most knowledge, while intermediate knowledge participants often failed to do so. This evidence for knowledge effects on SERP link selection may contribute to understanding the potential for personalization of search results ranking based on user domain knowledge.

© All rights reserved Cole et al. and/or ACM Press

Edit | Del

Oh, Kyong Eun and Belkin, Nicholas J. (2011): Cross analysis of keeping personal information in different forms. In: Proceedings of the 2011 iConference 2011. pp. 732-733.

This paper examines keeping behavior of personal information in different forms by reviewing and analyzing previous empirical studies on keeping personal information. By adopting user-centered and cross-form perspectives, similar behaviors as well as unique behaviors in keeping different forms of personal information are reviewed and comparatively analyzed. The identification of similarities and differences of keeping behaviors among different forms of personal information widens our understanding of personal information management (PIM). The results of this study have practical implication for human computer interaction (HCI) studies in designing tools, devices and interfaces that are more effective in supporting individuals' PIM.

© All rights reserved Oh and Belkin and/or ACM Press

Edit | Del

Liu, Jingjing, Cole, Michael J., Liu, Chang, Bierig, Ralf, Gwizdka, Jacek, Belkin, Nicholas J., Zhang, Jun and Zhang, Xiangmin (2010): Search behaviors in different task types. In: JCDL10 Proceedings of the 2010 Joint International Conference on Digital Libraries 2010. pp. 69-78.

Personalization of information retrieval tailors search towards individual users to meet their particular information needs by taking into account information about users and their contexts, often through implicit sources of evidence such as user behaviors. Task types have been shown to influence search behaviors including usefulness judgments. This paper reports on an investigation of user behaviors associated with different task types. Twenty-two undergraduate journalism students participated in a controlled lab experiment, each searching on four tasks which varied on four dimensions: complexity, task product, task goal and task level. Results indicate regular differences associated with different task characteristics in several search behaviors, including task completion time, decision time (the time taken to decide whether a document is useful or not), and eye fixations, etc. We suggest these behaviors can be used as implicit indicators of the user's task type.

© All rights reserved Liu et al. and/or their publisher

Edit | Del

Liu, Jingjing and Belkin, Nicholas J. (2010): Personalizing information retrieval for people with different levels of topic knowledge. In: JCDL10 Proceedings of the 2010 Joint International Conference on Digital Libraries 2010. pp. 383-384.

Edit | Del

Liu, Jingjing and Belkin, Nicholas J. (2010): Personalizing information retrieval for multi-session tasks: the roles of task stage and task type. In: Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2010. pp. 26-33.

Dwell time as a user behavior has been found in previous studies to be an unreliable predictor of document usefulness, with contextual factors such as the user's task needing to be considered in its interpretation. Task stage has been shown to influence search behaviors including usefulness judgments, as has task type. This paper reports on an investigation of how task stage and task type may help predict usefulness from the time that users spend on retrieved documents, over the course of several information seeking episodes. A 3-stage controlled experiment was conducted with 24 participants, each coming 3 times to work on 3 sub-tasks of a general task, couched either as "parallel" or "dependent" task type. The full task was to write a report on the general topic, with interim documents produced for each sub-task. Results show that task stage can help in inferring document usefulness from decision time, especially in the parallel task. The findings can be used to increase accuracy in predicting document usefulness and accordingly in personalizing search for multi-session tasks.

© All rights reserved Liu and Belkin and/or their publisher

Edit | Del

Liu, Jingjing, Liu, Chang, Gwizdka, Jacek and Belkin, Nicholas J. (2010): Can search systems detect users' task difficulty?: some behavioral signals. In: Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2010. pp. 845-846.

In this paper, we report findings on how user behaviors vary in tasks with different difficulty levels as well as of different types. Two behavioral signals: document dwell time and number of content pages viewed per query, were found to be able to help the system detect when users are working with difficult tasks.

© All rights reserved Liu et al. and/or their publisher

Edit | Del

Bierig, Ralf, Cole, Michael J., Gwizdka, Jacek and Belkin, Nicholas J. (2010): A Data Analysis and Modelling Framework for the Evaluation of Interactive Information Retrieval. In: Gurrin, Cathal, He, Yulan, Kazai, Gabriella, Kruschwitz, Udo, Little, Suzanne, Roelleke, Thomas, Rger, Stefan M. and Rijsbergen, Keith van (eds.) Advances in Information Retrieval - 32nd European Conference on IR Research - ECIR 2010 March 28-31, 2010, Milton Keynes, UK. pp. 673-674.

Edit | Del

Cole, Michael J., Gwizdka, Jacek, Bierig, Ralf, Belkin, Nicholas J., Liu, Jingjing, Liu, Chang and Zhang, Xiangmin (2010): Linking search tasks with low-level eye movement patterns. In: Proceedings of the 2010 Annual European Conference on Cognitive Ergonomics 2010. pp. 109-116.

Motivation -- On-the-task detection of the task type and task attributes can benefit personalization and adaptation of information systems. Research approach -- A web-based information search experiment was conducted with 32 participants using a multi-stream logging system. The realistic tasks were related directly to the backgrounds of the participants and were of distinct task types. Findings/Design -- We report on a relationship between task and individual reading behaviour. Specifically we show that transitions between scanning and reading behaviour in eye movement patterns are an implicit indicator of the current task. Research limitations/Implications -- This work suggests it is plausible to infer the type of information task from eye movement patterns. One limitation is a lack of knowledge about the general reading model differences across different types of tasks in the population. Although this is an experimental study we argue it can be generalized to real world text-oriented information search tasks. Originality/Value -- This research presents a new methodology to model user information search task behaviour. It suggests promise for detection of information task type based on patterns of eye movements. Take away message -- With increasingly complex computer interaction, knowledge about the type of information task can be valuable for system personalization. Modelling the reading/scanning patterns of eye movements can allow inference about the task type and task attributes.

© All rights reserved Cole et al. and/or their publisher

Edit | Del

Shen, Xuehua, Zhai, Chengxiang and Belkin, Nicholas J. (2009): Adaptive Clustering of Search Results. In: Proceedings of the 2009 Conference on User Modeling, Adaptation and Personalization 2009. pp. 447-453.

Clustering of search results has been shown to be advantageous over the simple list presentation of search results. However, in most clustering interfaces, the clusters are not adaptive to a user's interaction with the clustering results, and the important question "how to optimize the benefit of a clustering interface for a user" has not been well addressed in the previous work. In this paper, we study how to exploit a user's clickthrough information to adaptively reorganize the clustering results and help a user find the relevant information more quickly. We propose four strategies for adapting clustering results based on user actions. We propose a general method to simulate different kinds of users and linearize the cluster results so that we can compute regular retrieval measures. The simulation experiments show that the adaptation strategies have different performance for different types of users; in particular, they are effective for "smart users" who can correctly recognize the best clusters, but not effective for "dummy users" who follow system's ranking of results. We further conduct a user study on one of the four adaptive clustering strategies to see if an adaptive clustering system using such a strategy can bring users better search experience than a static clustering system. The results show that there is generally no significant difference between the two systems from a user's perspective.

© All rights reserved Shen et al. and/or their publisher

Edit | Del

Belkin, Nicholas J. (2008): Some(What) Grand Challenges for Information Retrieval. In: Macdonald, Craig, Ounis, Iadh, Plachouras, Vassilis, Ruthven, Ian and White, Ryen W. (eds.) Advances in Information Retrieval - 30th European Conference on IR Research - ECIR 2008 March 30-April 3, 2008, Glasgow, UK. p. 1.

Edit | Del

Yuan, Xiaojun and Belkin, Nicholas J. (2007): Supporting multiple information-seeking strategies in a single system framework. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2007. pp. 247-254.

This paper reports on an experiment comparing the retrieval effectiveness of an interactive information retrieval (IIR) system which adapts to support different information seeking strategies, with that of a standard baseline IIR system. The experiment, with 32 subjects each searching on 8 different topics, indicates that using the integrated IIR system resulted in significantly better performance, including user satisfaction with search results, significantly more effective interaction, and significantly better usability than using the baseline system.

© All rights reserved Yuan and Belkin and/or ACM Press

Edit | Del

Muresan, Gheorghe, Cole, Michael, Smith, Catherine L., Liu, Lu and Belkin, Nicholas J. (2006): Does Familiarity Breed Content? Taking Account of Familiarity with a Topic in Personalizing Information Retrieval. In: HICSS 2006 - 39th Hawaii International International Conference on Systems Science 4-7 January, 2006, Kauai, HI, USA. .

Edit | Del

Muresan, Gheorghe, Smith, Catherine L., Cole, Michael, Liu, Lu and Belkin, Nicholas J. (2006): Detecting Document Genre for Personalization of Information Retrieval. In: HICSS 2006 - 39th Hawaii International International Conference on Systems Science 4-7 January, 2006, Kauai, HI, USA. .

Edit | Del

Lin, Shin-jeng and Belkin, Nicholas J. (2005): Validation of a model of information seeking over multiple search sessions. In JASIST - Journal of the American Society for Information Science and Technology, 56 (4) pp. 393-415.

Edit | Del

Kelly, Diane and Belkin, Nicholas J. (2004): Display time as implicit feedback: understanding task effects. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2004. pp. 377-384.

Recent research has had some success using the length of time a user displays a document in their web browser as implicit feedback for document preference. However, most studies have been confined to specific search domains, such as news, and have not considered the effects of task on display time, and the potential impact of this relationship on the effectiveness of display time as implicit feedback. We describe the results of an intensive naturalistic study of the online information-seeking behaviors of seven subjects during a fourteen-week period. Throughout the study, subjects' online information-seeking activities were monitored with various pieces of logging and evaluation software. Subjects were asked to identify the tasks with which they were working, classify the documents that they viewed according to these tasks, and evaluate the usefulness of the documents. Results of a user-centered analysis demonstrate no general, direct relationship between display time and usefulness, and that display times differ significantly according to specific task, and according to specific user.

© All rights reserved Kelly and Belkin and/or ACM Press

Edit | Del

Wu, Mingfang, Muresan, Gheorghe, McLean, Alistair, Tang, Muh-Chyun (Morris), Wilkinson, Ross, Li, Yuelin, Lee, Hyuk-Jin and Belkin, Nicholas J. (2004): Human versus machine in the topic distillation task. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2004. pp. 385-392.

This paper reports on and discusses a set of user experiments using the TREC 2003 Web interactive track protocol. The focus is on comparing humans and machine algorithms in terms of performance in a topic distillation task. We also investigated the effect of the search results layout in supporting the users' effort. We have demonstrated that machines can perform nearly as well as people on the topic distillation task. Given a system tailored to the task there is significant performance improvement and finally, given a presentation that supports the task, there is strong user satisfaction.

© All rights reserved Wu et al. and/or ACM Press

Edit | Del

Kelly, Diane, Diaz, Fernando, Belkin, Nicholas J. and Allan, James (2004): A User-Centered Approach to Evaluating Topic Models. In: Mcdonald, Sharon and Tait, John (eds.) Advances in Information Retrieval - 26th European Conference on IR Research - ECIR 2004 April 5-7, 2004, Sunderland, UK. pp. 27-41.

Edit | Del

Belkin, Nicholas J., Kelly, D., Kim, G., Kim, J.-Y., Lee, H.-J., Muresan, G., Tang, M.-C., Yuan, X.-J. and Cool, C. (2003): Query length in interactive information retrieval. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2003. pp. 205-212.

Query length in best-match information retrieval (IR) systems is well known to be positively related to effectiveness in the IR task, when measured in experimental, non-interactive environments. However, in operational, interactive IR systems, query length is quite typically very short, on the order of two to three words. We report on a study which tested the effectiveness of a particular query elicitation technique in increasing initial searcher query length, and which tested the effectiveness of queries elicited using this technique, and the relationship in general between query length and search effectiveness in interactive IR. Results show that the specific technique results in longer queries than a standard query elicitation technique, that this technique is indeed usable, that the technique results in increased user satisfaction with the search, and that query length is positively correlated with user satisfaction with the search.

© All rights reserved Belkin et al. and/or ACM Press

Edit | Del

Yuan, Xiao-Jun, Belkin, Nicholas J. and Kim, Ja-Young (2002): The relationship between ASK and relevance criteria. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2002. pp. 359-360.

Edit | Del

Kelly, Diane, Yuan, Xiao-Jun, Belkin, Nicholas J., Murdock, Vanessa and Croft, W. Bruce (2002): Features of documents relevant to task- and fact-oriented questions. In: Proceedings of the 2002 ACM CIKM International Conference on Information and Knowledge Management November 4-9, 2002, McLean, VA, USA. pp. 645-647.

Edit | Del

Belkin, Nicholas J. (2000): The human element: helping people find what they don't know. In Communications of the ACM, 43 (8) pp. 58-61.

Edit | Del

Golovchinsky, Gene and Belkin, Nicholas J. (1999): Innovation and Evaluation of Information Exploration Interfaces: A CHI98 Workshop. In ACM SIGCHI Bulletin, 31 (1) pp. 22-25.

This report summarizes a workshop held at CHI 98 that focused on several aspects of information exploration, including user interfaces, theory, and evaluation. Information exploration is a common activity that spans a variety of media and is an integral component of many information seeking behaviors that people engage in. The complexity of this activity, and the need to support it appropriately, led us to propose this workshop. Over the course of two days, we examined several aspects of this problem, struggled with a few definitions, and came away with a better understanding of the design space. Here we summarize those efforts.

© All rights reserved Golovchinsky and Belkin and/or ACM Press

Edit | Del

Savage-Knepshield, Pamela A. and Belkin, Nicholas J. (1999): Interaction in Information Retrieval: Trends Over Time. In JASIST - Journal of the American Society for Information Science and Technology, 50 (12) pp. 1067-1082.

Edit | Del

Savage, Pamela, Belkin, Nicholas J., Cool, Colleen and Xie, Hong (1997): An Investigation of Mental Models and Information Seeking Behavior in a Novel Task. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1997. p. 340.

Mental models are representations of objects, events, and processes that people construct through interaction with their environments. As people interact with an information retrieval (IR) system, they infer how that system works and they develop a set of expectations that they use to guide their future interactions with that system. Often these models may be applied to other IR systems, domains, and novel tasks to varying degrees of success. We present data that describe the interactive searching behavior of twelve searchers using the INQUERY retrieval engine in the context of the TREC-5 interactive task. Our pre-search interview, in which participants described the methods that they would use when conducting an online search in order to identify as many "aspects" as possible for a topic, was designed to elicit users' mental models. Based upon a content analysis of the responses, we were able to derive a classification scheme comprised of three mental models employed by our participants: 1) start with anything, evaluate results, then plan the search, 2) start with general concepts then go to specific terms, and 3) identify keywords and try them one at a time. We discuss the mental models held by our experienced searchers, how they corresponded to actual searching behavior, the extent to which users' models "fit" the novel aspects task, the relative benefits and limitations of the models impact on retrieval performance, and implications for system design.

© All rights reserved Savage et al. and/or ACM Press

Edit | Del

Koenemann, Jurgen and Belkin, Nicholas J. (1996): A Case for Interaction: A Study of Interactive Information Retrieval Behavior and Effectiveness. In: Tauber, Michael J., Bellotti, Victoria, Jeffries, Robin, Mackinlay, Jock D. and Nielsen, Jakob (eds.) Proceedings of the ACM CHI 96 Human Factors in Computing Systems Conference April 14-18, 1996, Vancouver, Canada. pp. 205-212.

This study investigates the use and effectiveness of an advanced information retrieval (IR) system (INQUERY). 64 novice IR system users were studied in their use of a baseline version of INQUERY compared with one of three experimental versions, each offering a different level of interaction with a relevance feedback facility for automatic query reformulation. Results, in an information filtering task, indicate that: these subjects, after minimal training, were able to use the baseline system reasonably effectively; availability and use of relevance feedback increased retrieval effectiveness; and increased opportunity for user interaction with and control of relevance feedback made the interactions more efficient and usable while maintaining or increasing effectiveness.

© All rights reserved Koenemann and Belkin and/or ACM Press

Edit | Del

Veerasamy, Aravindan and Belkin, Nicholas J. (1996): Evaluation of a Tool for Visualization of Information Retrieval Results. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1996. pp. 85-92.

We report on the design and evaluation of a visualization tool for Information Retrieval (IR) systems that aims to help the end user in the following respects: * As an indicator of document relevance, the tool graphically provides specific query related information about individual documents * As a diagnosis tool, it graphically provides aggregate information about the query results that could help in identifying how the different query terms influence the retrieval and ranking of documents. Two different experiments using TREC-4 data were conducted to evaluate the effectiveness of this tool. Results, while mixed, indicate that visualization of this sort may provide useful support for judging the relevance of documents, in particular by enabling users to make more accurate decisions about which documents to inspect in detail. Problems in evaluation of such tools in interactive environments are discussed.

© All rights reserved Veerasamy and Belkin and/or ACM Press

Edit | Del

Cool, Colleen, Belkin, Nicholas J. and Koenemann, Jurgen (1996): On the Potential Utility of Negative Relevance Feedback in Interactive Information Retrieval. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1996. p. 341.

Automatic relevance feedback (RF) is gradually being incorporated into interactive information retrieval (IR) systems. It is generally the case that such systems use only positive, and not negative, relevance judgements for query modification. We present results of an empirical study of online searchers' relevance judgement behaviors which indicate that users of IR systems could and would make use of negative RF if it were offered. The data we present were collected as part of a larger study done within the context of TREC-4. Fifty volunteer searchers were recruited to search for two TREC topics each, using the INQUERY IR system, with automatic RF available. We collected the following types of data in our study: pre- and post-search interviews; videotapes of the searches, including "thinking aloud" protocols; and search logs. During the post-search interview, searchers discussed their uses of and reactions to automatic RF. Our analysis of the post-search interview and verbal protocol data identified four functions of negative RF desired by users in our study, and the types of searching problems addressed by each. These data also suggest a specific, novel method of implementing negative RF; that is, to expand queries by incorporating terms, with negative weights, which appear only in negatively judged documents.

© All rights reserved Cool et al. and/or ACM Press

Edit | Del

Taghva, Kazem, Fox, Edward, Robertson, Stephen, Belkin, Nicholas J., Lewis, David and Harman, Donna (1995): Education for IR. In: Proceedings of the Eighteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1995. p. 358.

The SIGIR Education Committee has recently been formed based on the model of the SIGCHI committee which completed its final report in 1992. The committee is charged with developing curriculum recommendations for IR-related education to serve the computer science, library science, and information science communities. It will be soliciting input from information retrieval educators and the consumers of IR education (students and employers), with a view of determining the current status of IR education, the marketplace, and future direction. The committee is also interested in: * clearinghouses for IR courseware and training materials * electronic as well as traditional courses * demonstrations for online access to state of the art systems * other innovative efforts The purpose of this panel is to report briefly on the activities of the Education Committee and to stimulate discussion on the state of information retrieval education. The panel will consist of three IR educators from different communities (computer science, library science, information science) who will give brief (about 10 minute) presentations on their view of the purpose and content of IR curricula; a representative from the government sector will report on the role of IR in government agencies; and a participant from the industrial sector will consider the role of IR education in industry.

© All rights reserved Taghva et al. and/or ACM Press

Edit | Del

Dumais, Susan, Belkin, Nicholas J., Borgman, Christine L. and Hancock-Beaulieu, Micheline (1994): Evaluating Interactive Retrieval Systems. In: Proceedings of the Seventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1994. p. 361.

Most current information retrieval systems are highly interactive. Users ask queries, get immediate feedback, refine their queries, and so on. Methods for evaluating these dynamic systems have not kept pace with the rapid advances in system design. It is no longer enough to use the standard precision-recall measures to evaluate and to improve interactive retrieval systems. There is often no single final query to evaluate, with useful information being gathered from many different queries along the way. In addition, interfaces play a critical role in building effective retrieval systems. The best retrieval algorithm can be rendered functionally useless if the interface to it is unusable. Conversely, of course, the spiffiest new interface is not worth much without a good retrieval engine behind it. It would be easy if one could study interfaces and retrieval engines separately and take the best of both worlds. Unfortunately, there are important interactions that cannot be evaluated by studying components in isolation -- e.g., how do you incorporate ranking or relevance feedback for a Boolean retrieval engine, or how do you highlight matching terms if complex syntactic and semantic processing of queries is used? The design of effective interactive retrieval environments will require careful attention to the larger human - interface - retrieval-engine system. Systematic, generalizable evaluations of these larger interactive systems are possible both in the laboratory and in the field. The panelists will describe interactive retrieval experiments and experiences, focusing on: a) why it is important to study interactions b) how interactive retrieval performance should be measured, and c) how the methods for evaluation and findings generalize to other systems. Belkin will begin with an overview of some of the problems in evaluating interactive retrieval systems and will present a new framework characterizing IR as interaction with text. The remaining talks will describe end-user experiments involving highly interactive retrieval systems. The focus of these talks will be on the approaches, methods and instruments used to evaluate retrieval effectiveness and ease of use as well as the relationship between system functionality and the interface. Dumais will describe the importance of interfaces in retrieval, and will present examples of successful iterative interface design with the SuperBook and X-LSI systems. Hancock-Beaulieu will review a series of experiments on the Okapi system to systematically examine the effectiveness of different retrieval aids. Bergman will describe the multiple evaluation methods employed to study children's information-seeking behavior using the Science Library Catalog, a graphical browsing system supplemented by keyword searching tailored to children's skills, and attempts to generalize these evaluation methods to other IR environments.

© All rights reserved Dumais et al. and/or ACM Press

Edit | Del

Belkin, Nicholas J., Cool, C., Croft, W. Bruce and Callan, J. P. (1993): The Effect of Multiple Query Representations on Information Retrieval System Performance. In: Proceedings of the Sixteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1993. pp. 339-346.

Five independently generated Boolean query formulations for ten different TREC topics were produced by ten different expert online searchers. These different formulations were grouped, and the groups, and combinations of them, were used as searches against the TREC test collection, using the INQUERY probabilistic inference network retrieval engine. Results show that progressive combination of query formulations leads to progressively improving retrieval performance. Results were compared against the performance of INQUERY natural language based queries, and in combination with them. The issue of recall as a performance measure in large databases was raised, since overlap between the searches conducted in this study, and the TREC-1 searches, was smaller than expected.

© All rights reserved Belkin et al. and/or ACM Press

Edit | Del

Marchetti, P. G., Vazzana, S., Panero, R. and Belkin, Nicholas J. (1993): BRAQUE: An Interface to Support Browsing and Interactive Query Formulation in Information Retrieval Systems. In: Proceedings of the Sixteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1993. p. 358.

BRAQUE [1,2] is a Windows-based interface to a large-scale operational information retrieval system. The general design of BRAQUE is based on two concepts. One is that of supporting end users of information retrieval systems in a variety of information-seeking strategies, with the user being able to move easily and effectively from any one such strategy to any other. Included in this concept is the idea that information retrieval is an inherently interactive process, and that support of users should be support of their interaction, with all of the system resources. The second concept is that of a two-level hypertext model of information retrieval system databases, which provides a framework in which the kind of interactive support we propose can be implemented. The current version of BRAQUE is a working prototype interface to the ESA-QUEST system. It provides facilities for browsing in thesaural structures, and among documents and terms related to documents. It supports information retrieval without query formulation, and it supports progressive, interactive query formulation and reformulation, as well as general browsing and exploration.

© All rights reserved Marchetti et al. and/or ACM Press

Edit | Del

Tague-Sutcliffe, Jean and Belkin, Nicholas J. (1992): B. C. Bertie Brookes, 1910-1991. In JASIST - Journal of the American Society for Information Science and Technology, 43 (4) pp. 273-275.

Edit | Del

Belkin, Nicholas J. and Croft, W. Bruce (1992): Information Filtering and Information Retrieval: Two Sides of the Same Coin?. In Communications of the ACM, 35 (12) pp. 29-38.

Edit | Del

Belkin, Nicholas J. (1991): A Methodology for Taking Account of User Tasks, Goals and Behavior for Design of Computerized Library Catalogs. In ACM SIGCHI Bulletin, 23 (1) pp. 61-65.

Edit | Del

Belkin, Nicholas J. (1991): B. C. Brookes: In Memoriam. In: Proceedings of the Fourteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1991. p. 1.

Edit | Del

Croft, W. Bruce, Belkin, Nicholas J., Bruandet, Marie-France, Kuhlen, Rainer and Oren, Tim (1990): Hypertext and Information Retrieval: What are the Fundamental Concepts?. In: Rizk, Antoine, Streitz, Norbert A. and Andre, Jacques (eds.) ECHT 90 - European Conference on Hypertext November 27-30, 1990, Versailles, France. pp. 362-366.

Both hypertext and information retrieval (IR) systems provide access to databases consisting primarily of text documents. Both types of systems structure the content of these documents and support interaction with the users in order to improve the effectiveness of retrieval. Despite these similarities, hypertext and IR are generally regarded as separate research areas, with some overlap, but essentially different research agendas. To clarify these differences as well as the areas of overlap, the members of this panel will attempt to define the fundamental concepts and the major research issues in each area, with special emphasis on their own research.

© All rights reserved Croft et al. and/or Cambridge University Press

Edit | Del

Croft, Bruce, Belkin, Nicholas J., Bikson, Tora, Malone, Thomas W. and Marchionini, Gary (1990): Evaluating Computer-Based Tools in Organizations. In: Lochovsky, Frederick H. and Allen, Robert (eds.) Proceedings of the Conference on Office Information Systems 1990 April 25-27, 1990, Cambridge, Massachusetts, USA. p. 193.

Most of the computer systems that are proposed and implemented in the OIS area are very difficult to evaluate, both in terms of the effectiveness of the functions they perform and their impact on organizations. The time and space efficiency of a tool is readily measured, but this may be only a small part of a tool's "performance". Researchers in areas such as CSCW, Information Retrieval, Hypertext, and User Interfaces have been forced to confront the issue of evaluation in order to compare systems and determine when progress has been made. There is, however, still much debate about the correct methodologies for evaluating such complex, interactive systems. System developers are also concerned with measuring the "usability" of their systems and designing systems iteratively based on user feedback. It is not clear whether these approaches compete with, or are complementary to more traditional experimental approaches. The panelists come from a variety of backgrounds and have each had experience evaluating complex systems. By comparing their experiences and the techniques used in their areas, we hope to be able to make some progress on the issue of how systems can be evaluated and compared.

© All rights reserved Croft et al. and/or ACM Press

Edit | Del

Belkin, Nicholas J. and Marchetti, Pier Giorgio (1990): Determining the Functionality and Features of an Intelligent Interface to an Information Retrieval System. In: Proceedings of the Thirteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1990. pp. 151-177.

In this paper, we propose a method for specifying the functionality of an intelligent interface to large-scale information retrieval systems and for implementing those functions in an operational environment. The method is based on a progressive, three-stage model of intelligent information support; a high-level cognitive task analysis of the information retrieval problem; a low-level specification of the host system functionality; and, derivation of explicit relations between the system functions and the cognitive tasks. This method is applied, by example, in the context of the European Space Agency Information Retrieval Service, with some specific suggestions for implementation of a stage one intelligent interface to that system.

© All rights reserved Belkin and Marchetti and/or ACM Press

Edit | Del

Frakes, W. B., Belkin, Nicholas J., Prieto-Diaz, R. and Wartik, S. (1989): Information Retrieval and Software Reuse. In: Proceedings of the Twelfth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1989. pp. 251-256.

Software reuse is widely believed to be the most promising technology for improving software quality and productivity. There are many technical and non-technical problems to be solved, however, before widespread reuse of software lifecycle objects becomes a reality. One class of problem concerns the classification, storage, and retrieval of reusable components. Panel members will discuss these problems and some approaches to solving them.

© All rights reserved Frakes et al. and/or ACM Press

Edit | Del

Borgman, Christine L., Belkin, Nicholas J., Croft, W. Bruce, Lesk, Michael E. and Landauer, Thomas K. (1988): Retrieval Systems for the Information Seeker: Can the Role of the Intermediary be Automated?. In: Soloway, Elliot, Frye, Douglas and Sheppard, Sylvia B. (eds.) Proceedings of the ACM CHI 88 Human Factors in Computing Systems Conference June 15-19, 1988, Washington, DC, USA. pp. 51-53.

The introduction of automated information retrieval (IR) systems was met with great enthusiasm and predictions that manual literature searching soon would be replaced. Three decades later, IR systems have not progressed to the stage where any but the dedicated few can operate them without a highly skilled human intermediary acting as interface between user and system. In the interim, we have learned that the retrieval process is extremely complex both in terms of understanding people and their communication and in terms of understanding scientific information and technical vocabulary. Experiments with new techniques suggest to many the possibility of eliminating the human intermediary, either in large part or altogether; others would argue that the retrieval problems are too complex to be resolved for more than highly restricted domains. The possibility of eliminating the human intermediary is of current research interest to the several disciplines that are represented on this panel.

© All rights reserved Borgman et al. and/or ACM Press

Edit | Del

Belkin, Nicholas J. (1988): On the Nature and Function of Explanation in Intelligent Information Retrieval. In: Proceedings of the Eleventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1988. pp. 135-145.

We discuss the complexity of explanation activity in human goal-directed dialogue, and suggest that this complexity ought to be taken account of in the design of explanation in human-computer interaction. We propose a general model of clarity in human-computer systems, of which explanation is one component. On the bases of: this model; of a model of human-intermediary interaction in the document retrieval situation as one of cooperative model-building for the purpose of developing an appropriate search formulation; and, on the results of empirical observation of human user-human intermediary interaction in information systems, we propose a model for explanation by the computer intermediary in information retrieval.

© All rights reserved Belkin and/or ACM Press

Edit | Del

Belkin, Nicholas J., Brooks, H. M. and Daniels, P. J. (1987): Knowledge Elicitation Using Discourse Analysis. In International Journal of Man-Machine Studies, 27 (2) pp. 127-144.

This paper is concerned with the use of discourse analysis and observation to elicit expert knowledge. In particular, we describe the use of these techniques to acquire knowledge about expert problem solving in an information provision environment. Our method of analysis has been to make audio-recordings of real-life information interactions between users (the clients) and human intermediaries (the experts) in document retrieval situations. These tapes have then been transcribed and analysed utterance-by-utterance in the following ways: assigning utterances to one of the prespecified functional categories; identifying the specific purposes of each utterance; determining the knowledge required to perform each utterance; grouping utterances into functional and focus-based sequences. The long-term goal of the project is to develop an intelligent document retrieval system based on a distributed expert, blackboard architecture.

© All rights reserved Belkin et al. and/or Academic Press

Edit | Del

Belkin, Nicholas J. and Kwasnik, B. H. (1986): Using Structural Representations of Anomalous States of Knowledge for Choosing Document Retrieval Strategies. In: Proceedings of the Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1986. pp. 11-22.

We report on a project which attempts to classify representations of the anomalous states of knowledge (ASKs) of users of document retrieval systems on the basis of structural characteristics of the representations, and which specifies different retrieval strategies and ranking mechanisms for each ASK class. The classification and retrieval strategy specification is based on 53 real problem statements, 35 of which have a total of 250 evaluated documents. Four facets of the ASK structures have been tentatively identified, whose combinations determine the method and order of application of five basic ranking strategies. This work is still in progress, so results presented here are incomplete.

© All rights reserved Belkin and Kwasnik and/or ACM Press

Add publication
Show list on your website

Join our community and advance:




Join our community!

Page Information

Page maintainer: The Editorial Team