Number of co-authors:89
Number of publications with 3 favourite co-authors:Joseph A. Konstan:19Shyong K. Lam:9Shilad Sen:9
John Riedl's 3 most productive colleagues in number of publications:Robert E. Kraut:98Loren Terveen:69Mark S. Ackerman:67
go to course
Emotional Design: How to make products people will love
Starts the day after tomorrow !
go to course
UI Design Patterns for Successful Software
85% booked. Starts in 10 days
Marc Hassenzahl explains the fascinating concept of User Experience and Experience Design. Commentaries by Don Norman, Eric Reiss, Mark Blythe, and Whitney Hess
User Experience and Experience Design !
Our Latest Books
The Social Design of Technical Systems: Building technologies for communities. 2nd Edition
by Brian Whitworth and Adnan Ahmad
Gamification at Work: Designing Engaging Business Software
by Janaki Mythily Kumar and Mario Herger
The Social Design of Technical Systems: Building technologies for communities
by Brian Whitworth and Adnan Ahmad
The Encyclopedia of Human-Computer Interaction, 2nd Ed.
by Mads Soegaard and Rikke Friis Dam
Has also published under the name of:
"J. Riedl" and "John T. Riedl"
Personal Homepage: http://www-users.cs.umn.edu/~riedl/
My research focus is on collaborative systems that support human interaction through computer systems. My career goal is to understand how to develop and apply computer technology to the problems of human organizations.
One of the biggest such problems is getting the right information to the right people. The Internet has democratized the publishing process. Now, anyone who wants can publish anything they want, just by creating a Web site. We humans are hopelessly overmatched by the increasing volumes of information that are published. Collaborative filtering is a technology that enables us to all work together to sift through the millions of documents on any topic to find those that are most appropriate for each of us. Collaborative filtering works by learning which kinds of documents each of us likes, and finding other people who share out interests.
Across our entire research program, our goal is to understand how computers can be used to help people process information more efficiently, and work together better.
I am currently involved in several research projects to explore these topics.
Publications by John Riedl (bibliography)
Wang, Loxley Sijia, Chen, Jilin, Ren, Yuqing and Riedl, John (2012): Searching for the Goldilocks zone: trade-offs in managing online volunteer groups. In: Proceedings of ACM CSCW12 Conference on Computer-Supported Cooperative Work 2012. pp. 989-998. Available online
Dedicated and productive members who actively contribute to community efforts are crucial to the success of online volunteer groups such as Wikipedia. What predicts member productivity? Do productive members stay longer? How does involvement in multiple projects affect member contribution to the community? In this paper, we analyze data from 648 WikiProjects to address these questions. Our results reveal two critical trade-offs in managing online volunteer groups. First, factors that increase member productivity, measured by the number of edits on Wikipedia articles, also increase likelihood of withdrawal from contributing, perhaps due to feelings of mission accomplished or burnout. Second, individual membership in multiple projects has mixed effects. It decreases the amount of work editors contribute to both the individual projects and Wikipedia as a whole. It increases withdrawal for each individual project yet reduces withdrawal from Wikipedia. We discuss how our findings expand existing theories to fit the online context and inform the design of new tools to improve online volunteer work.
© All rights reserved Wang et al. and/or ACM Press
Forte, Andrea, Antin, Judd, Bardzell, Shaowen, Honeywell, Leigh, Riedl, John and Stierch, Sarah (2012): Some of all human knowledge: gender and participation in peer production. In: Companion Proceedings of ACM CSCW12 Conference on Computer-Supported Cooperative Work 2012. pp. 33-36. Available online
The promise of peer production includes resources produced by volunteers and released freely for the world to use. Wikipedia and Open Source Software are famous examples of peer-produced projects. Anyone is free to participate, but not everybody does. Wikipedia aims to collect the "sum of all human knowledge", but only about 13% of editors on the site are female . In Open Source Software, the percentage of female contributors has been estimated near 1% . If women are not well represented among authors of the most widely accessed reference source on the planet, are important voices muted? Could these projects be even more impactful with more female participation? This panel includes experts in gender theory and open collaboration, activists, and representatives from peer-produced projects to discuss recent findings and trends in this complex and often contentious research space.
© All rights reserved Forte et al. and/or ACM Press
Kluver, Daniel, Nguyen, Tien T., Ekstrand, Michael, Sen, Shilad and Riedl, John (2012): How many bits per rating?. In: Proceedings of the 2012 ACM Conference on Recommender Systems 2012. pp. 99-106. Available online
Most recommender systems assume user ratings accurately represent user preferences. However, prior research shows that user ratings are imperfect and noisy. Moreover, this noise limits the measurable predictive power of any recommender system. We propose an information theoretic framework for quantifying the preference information contained in ratings and predictions. We computationally explore the properties of our model and apply our framework to estimate the efficiency of different rating scales for real world datasets. We then estimate how the amount of information predictions give to users is related to the scale ratings are collected on. Our findings suggest a tradeoff in rating scale granularity: while previous research indicates that coarse scales (such as thumbs up / thumbs down) take less time, we find that ratings with these scales provide less predictive value to users. We introduce a new measure, preference bits per second, to quantitatively reconcile this tradeoff.
© All rights reserved Kluver et al. and/or ACM Press
Ekstrand, Michael and Riedl, John (2012): When recommenders fail: predicting recommender failure for algorithm selection and combination. In: Proceedings of the 2012 ACM Conference on Recommender Systems 2012. pp. 233-236. Available online
Hybrid recommender systems -- systems using multiple algorithms together to improve recommendation quality -- have been well-known for many years and have shown good performance in recent demonstrations such as the NetFlix Prize. Modern hybridization techniques, such as feature-weighted linear stacking, take advantage of the hypothesis that the relative performance of recommenders varies by circumstance and attempt to optimize each item score to maximize the strengths of the component recommenders. Less attention, however, has been paid to understanding what these strengths and failure modes are. Understanding what causes particular recommenders to fail will facilitate better selection of the component recommenders for future hybrid systems and a better understanding of how individual recommender personalities can be harnessed to improve the recommender user experience. We present an analysis of the predictions made by several well-known recommender algorithms on the MovieLens 10M data set, showing that for many cases in which one algorithm fails, there is another that will correctly predict the rating.
© All rights reserved Ekstrand and Riedl and/or ACM Press
Lam, Shyong (Tony) K., Uduwage, Anuradha, Dong, Zhenhua, Sen, Shilad, Musicant, David R., Terveen, Loren and Riedl, John (2011): WP:clubhouse?: an exploration of Wikipedia's gender imbalance. In: Proceedings of the 2011 International Symposium on Wikis and Open Collaboration 2011. pp. 1-10. Available online
Wikipedia has rapidly become an invaluable destination for millions of information-seeking users. However, media reports suggest an important challenge: only a small fraction of Wikipedia's legion of volunteer editors are female. In the current work, we present a scientific exploration of the gender imbalance in the English Wikipedia's population of editors. We look at the nature of the imbalance itself, its effects on the quality of the encyclopedia, and several conflict-related factors that may be contributing to the gender gap. Our findings confirm the presence of a large gender gap among editors and a corresponding gender-oriented disparity in the content of Wikipedia's articles. Further, we find evidence hinting at a culture that may be resistant to female participation.
© All rights reserved Lam et al. and/or ACM Press
Halfaker, Aaron, Song, Bryan, Stuart, D. Alex, Kittur, Aniket and Riedl, John (2011): NICE: social translucence through UI intervention. In: Proceedings of the 2011 International Symposium on Wikis and Open Collaboration 2011. pp. 101-104. Available online
Social production systems such as Wikipedia rely on attracting and motivating volunteer contributions to be successful. One strong demotivating factor can be when an editor's work is discarded, or "reverted", by others. In this paper we demonstrate evidence of this effect and design a novel interface aimed at improving communication between the reverting and reverted editors. We deployed the interface in a controlled experiment on the live Wikipedia site, and report on changes in the behavior of 487 contributors who were reverted by editors using our interface. Our results suggest that simple interface modifications (such as informing Wikipedians that the editor they are reverting is a newcomer) can have substantial positive effects in protecting against contribution loss in newcomers and improving the quality of work done by more experienced contributors.
© All rights reserved Halfaker et al. and/or ACM Press
Halfaker, Aaron, Kittur, Aniket and Riedl, John (2011): Don't bite the newbies: how reverts affect the quantity and quality of Wikipedia work. In: Proceedings of the 2011 International Symposium on Wikis and Open Collaboration 2011. pp. 163-172. Available online
Reverts are important to maintaining the quality of Wikipedia. They fix mistakes, repair vandalism, and help enforce policy. However, reverts can also be damaging, especially to the aspiring editor whose work they destroy. In this research we analyze 400,000 Wikipedia revisions to understand the effect that reverts had on editors. We seek to understand the extent to which they demotivate users, reducing the workforce of contributors, versus the extent to which they help users improve as encyclopedia editors. Overall we find that reverts are powerfully demotivating, but that their net influence is that more quality work is done in Wikipedia as a result of reverts than is lost by chasing editors away. However, we identify key conditions -- most specifically new editors being reverted by much more experienced editors -- under which reverts are particularly damaging. We propose that reducing the damage from reverts might be one effective path for Wikipedia to solve the newcomer retention problem.
© All rights reserved Halfaker et al. and/or ACM Press
Musicant, David R., Ren, Yuqing, Johnson, James A. and Riedl, John (2011): Mentoring in Wikipedia: a clash of cultures. In: Proceedings of the 2011 International Symposium on Wikis and Open Collaboration 2011. pp. 173-182. Available online
The continuous success of Wikipedia depends upon its capability to recruit and engage new editors, especially those with new knowledge and perspectives. Yet Wikipedia over the years has become a complicated bureaucracy that may be difficult for newcomers to navigate. Mentoring is a practice that has been widely used in offline organizations to help new members adjust to their roles. In this paper, we draw insights from the offline mentoring literature to analyze mentoring practices in Wikipedia and how they influence editor behaviors. Our quantitative analysis of the Adopt-a-user program shows mixed success of the program. Communication between adopters and adoptees is correlated with the amount of article editing done by adoptees shortly after adoption. Our qualitative analysis of the communication between adopters and adoptees suggests that several key functions of mentoring are missing or not fulfilled consistently. Most adopters focus on establishing their legitimacy rather than acting proactively to guide, protect, and support the long-term growth of adoptees. We conclude with recommendations of how Wikipedia mentoring programs can evolve to take advantage of offline best practices.
© All rights reserved Musicant et al. and/or ACM Press
Riedl, John and Smyth, Barry (2011): Introduction to special issue on recommender systems. In ACM Transactions on the Web, 5 (1) p. 1. Available online
Chen, Jilin, Ren, Yuqing and Riedl, John (2010): The effects of diversity on group productivity and member withdrawal in online volunteer groups. In: Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems 2010. pp. 821-830. Available online
The "wisdom of crowds" argument emphasizes the importance of diversity in online collaborations, such as open source projects and Wikipedia. However, decades of research on diversity in offline work groups have painted an inconclusive picture. On the one hand, the broader range of insights from a diverse group can lead to improved outcomes. On the other hand, individual differences can lead to conflict and diminished performance. In this paper, we examine the effects of group diversity on the amount of work accomplished and on member withdrawal behaviors in the context of WikiProjects. We find that increased diversity in experience with Wikipedia increases group productivity and decreases member withdrawal -- up to a point. Beyond that point, group productivity remains high, but members are more likely to withdraw. Strikingly, no such diminishing returns were observed for differences in member interest, which increases productivity and decreases member withdrawal in a linear fashion. Our results suggest that the low visibility of individual differences in online groups may allow them to harvest more of the benefits of diversity while bearing less of the cost. We discuss how our findings can inform further research of online collaboration.
© All rights reserved Chen et al. and/or their publisher
Vig, Jesse, Soukup, Matthew, Sen, Shilad and Riedl, John (2010): Tag expression: tagging with feeling. In: Proceedings of the 2010 ACM Symposium on User Interface Software and Technology 2010. pp. 323-332. Available online
In this paper we introduce tag expression, a novel form of preference elicitation that combines elements from tagging and rating systems. Tag expression enables users to apply affect to tags to indicate whether the tag describes a reason they like, dislike, or are neutral about a particular item. We present a user interface for applying affect to tags, as well as a technique for visualizing the overall community's affect. By analyzing 27,773 tag expressions from 553 users entered in a 3-month period, we empirically evaluate our design choices. We also present results of a survey of 97 users that explores users' motivations in tagging and measures user satisfaction with tag expression.
© All rights reserved Vig et al. and/or their publisher
Lam, Shyong K., Karim, Jawed and Riedl, John (2010): The effects of group composition on decision quality in a social production community. In: GROUP10 International Conference on Supporting Group Work 2010. pp. 55-64. Available online
Online social production communities allow efficient construction of valuable and high-quality information sources. To be successful, community members must be effective at collaboration, including making collective decisions in the presence of disagreement. We examined over 100,000 decisions made by small working groups in Wikipedia, and analyzed how decision quality in these online groups is influenced by four group composition factors: the size of the group, how members were invited to the group, the prior experience of group members, and apparent bias shown by the group administrator. Our findings lead us to recommendations for designers of social production communities.
© All rights reserved Lam et al. and/or their publisher
Lam, Shyong (Tony) K. and Riedl, John (2009): Is Wikipedia growing a longer tail?. In: GROUP09 - International Conference on Supporting Group Work 2009. pp. 105-114. Available online
Wikipedia has millions of articles, many of which receive little attention. One group of Wikipedians believes these obscure entries should be removed because they are uninteresting and neglected; these are the deletionists. Other Wikipedians disagree, arguing that this long tail of articles is precisely Wikipedia's advantage over other encyclopedias; these are the inclusionists. This paper looks at two overarching questions on the debate between deletionists and inclusionists: (1) What are the implications to the long tail of the evolving standards for article birth and death? (2) How is viewership affected by the decreasing notability of articles in the long tail? The answers to five detailed research questions that are inspired by these overarching questions should help better frame this debate and provide insight into how Wikipedia is evolving.
© All rights reserved Lam and Riedl and/or their publisher
Vig, Jesse, Sen, Shilad and Riedl, John (2009): Tagsplanations: explaining recommendations using tags. In: Proceedings of the 2009 International Conference on Intelligent User Interfaces 2009. pp. 47-56. Available online
While recommender systems tell users what items they might like, explanations of recommendations reveal why they might like them. Explanations provide many benefits, from improving user satisfaction to helping users make better decisions. This paper introduces tagsplanations, which are explanations based on community tags. Tagsplanations have two key components: tag relevance, the degree to which a tag describes an item, and tag preference, the user's sentiment toward a tag. We develop novel algorithms for estimating tag relevance and tag preference, and we conduct a user study exploring the roles of tag relevance and tag preference in promoting effective tagsplanations. We also examine which types of tags are most useful for tagsplanations.
© All rights reserved Vig et al. and/or their publisher
Sen, Shilad, Vig, Jesse and Riedl, John (2009): Learning to recognize valuable tags. In: Proceedings of the 2009 International Conference on Intelligent User Interfaces 2009. pp. 87-96. Available online
Many websites use tags as a mechanism for improving item metadata through collective user effort. Users of tagging systems often apply far more tags to an item than a system can display. These tags can range in quality from tags that capture a key facet of an item, to those that are subjective, irrelevant, or misleading. In this paper we explore tag selection algorithms that choose the tags that sites display. Based on 225,000 ratings and survey responses, we conduct offline analyses of 21 tag selection algorithms. We select the three best performing algorithms from our offline analysis, and deploy them live on the MovieLens website to 5,695 users for three months. Based on our results, we offer tagging system designers advice about tag selection algorithms.
© All rights reserved Sen et al. and/or their publisher
Sen, Shilad, Vig, Jesse and Riedl, John (2009): Tagommenders: connecting users to items through tags. In: Proceedings of the 2009 International Conference on the World Wide Web 2009. pp. 671-680. Available online
Tagging has emerged as a powerful mechanism that enables users to find, organize, and understand online entities. Recommender systems similarly enable users to efficiently navigate vast collections of items. Algorithms combining tags with recommenders may deliver both the automation inherent in recommenders, and the flexibility and conceptual comprehensibility inherent in tagging systems. In this paper we explore tagommenders, recommender algorithms that predict users' preferences for items based on their inferred preferences for tags. We describe tag preference inference algorithms based on users' interactions with tags and movies, and evaluate these algorithms based on tag preference ratings collected from 995 MovieLens users. We design and evaluate algorithms that predict users' ratings for movies based on their inferred tag preferences. Our tag-based algorithms generate better recommendation rankings than state-of-the-art algorithms, and they may lead to flexible recommender systems that leverage the characteristics of items users find most important.
© All rights reserved Sen et al. and/or ACM Press
Halfaker, Aaron, Kittur, Aniket, Kraut, Robert E. and Riedl, John (2009): A jury of your peers: quality, experience and ownership in Wikipedia. In: Proceedings of the 2009 International Symposium on Wikis 2009. p. 15. Available online
Wikipedia is a highly successful example of what mass collaboration in an informal peer review system can accomplish. In this paper, we examine the role that the quality of the contributions, the experience of the contributors and the ownership of the content play in the decisions over which contributions become part of Wikipedia and which ones are rejected by the community. We introduce and justify a versatile metric for automatically measuring the quality of a contribution. We find little evidence that experience helps contributors avoid rejection. In fact, as they gain experience, contributors are even more likely to have their work rejected. We also find strong evidence of ownership behaviors in practice despite the fact that ownership of content is discouraged within Wikipedia.
© All rights reserved Halfaker et al. and/or their publisher
Agrahri, Arun Kumar, Manickam, Divya Anand Thattandi and Riedl, John (2008): Can people collaborate to improve the relevance of search results?. In: Proceedings of the 2008 ACM Conference on Recommender Systems 2008. pp. 283-286. Available online
Search engines are among the most-used resources on the internet. However, even today's most successful search engines struggle to provide high quality search results. According to recent studies as many as 50 percent of web search sessions fail to find any relevant results for the searcher. Researchers have proposed social search techniques, in which early searchers provide feedback that is used to improve relevance for later searchers. In this paper we investigate foundational questions of social search. In particular, we directly assess the degree of agreement among users about the relevance ranking of search results. We developed a simulated search engine interface that systematically randomizes Google's normal relevance ordering of the items presented to users. Our results show that (a) people are biased toward items in the top of the search lists, even if the list is randomized; (b) people explicit feedback is not biased and (c) people's shared preferences do not always agree with Google's result order. These results suggest that social search techniques might improve the effectiveness of web search engines.
© All rights reserved Agrahri et al. and/or ACM Press
Riedl, John and Jameson, Anthony (2007): Advanced topics in recommendation. In: Proceedings of the 2007 International Conference on Intelligent User Interfaces 2007. p. 11. Available online
This full-day tutorial is designed to convey an up-to-date, active understanding of a representative set of current developments in recommender systems that will help the participants to conduct cutting-edge research and/or to work more effectively with the currently widespread recommendation technology.
© All rights reserved Riedl and Jameson and/or ACM Press
Harper, F. Maxwell, Frankowski, Dan, Drenner, Sara, Ren, Yuqing, Kiesler, Sara, Terveen, Loren, Kraut, Robert E. and Riedl, John (2007): Talk amongst yourselves: inviting users to participate in online conversations. In: Proceedings of the 2007 International Conference on Intelligent User Interfaces 2007. pp. 62-71. Available online
Many small online communities would benefit from increased diversity or activity in their membership. Some communities run the risk of dying out due to lack of participation. Others struggle to achieve the critical mass necessary for diverse and engaging conversation. But what tools are available to these communities to increase participation? Our goal in this research was to spark contributions to the movielens.org discussion forum, where only 2% of the members write posts. We developed personalized invitations, messages designed to entice users to visit or contribute to the forum. In two field experiments, we ask (1) if personalized invitations increase activity in a discussion forum, (2) how the choice of algorithm for intelligently choosing content to emphasize in the invitation affects participation, and (3) how the suggestion made to the user affects their willingness to act. We find that invitations lead to increased participation, as measured by levels of reading and posting. More surprisingly, we find that invitations emphasizing the social nature of the discussion forum increase user activity, while invitations emphasizing other details of the discussion are less successful.
© All rights reserved Harper et al. and/or ACM Press
Priedhorsky, Reid, Chen, Jilin, Lam, Shyong (Tony) K., Panciera, Katherine, Terveen, Loren and Riedl, John (2007): Creating, destroying, and restoring value in wikipedia. In: GROUP07: International Conference on Supporting Group Work 2007. pp. 259-268. Available online
Wikipedia's brilliance and curse is that any user can edit any of the encyclopedia entries. We introduce the notion of the impact of an edit, measured by the number of times the edited version is viewed. Using several datasets, including recent logs of all article views, we show that an overwhelming majority of the viewed words were written by frequent editors and that this majority is increasing. Similarly, using the same impact measure, we show that the probability of a typical article view being damaged is small but increasing, and we present empirically grounded classes of damage. Finally, we make policy recommendations for Wikipedia and other wikis in light of these findings.
© All rights reserved Priedhorsky et al. and/or ACM Press
Sen, Shilad, Harper, F. Maxwell, LaPitz, Adam and Riedl, John (2007): The quest for quality tags. In: GROUP07: International Conference on Supporting Group Work 2007. pp. 361-370. Available online
Many online communities use tags -- community selected words or phrases -- to help people find what they desire. The quality of tags varies widely, from tags that capture a key dimension of an entity to those that are profane, useless, or unintelligible. Tagging systems must often select a subset of available tags to display to users due to limited screen space. Because users often spread tags they have seen, selecting good tags not only improves an individual's view of tags, it also encourages them to create better tags in the future. We explore implicit (behavioral) and explicit (rating) mechanisms for determining tag quality. Based on 102,056 tag ratings and survey responses collected from 1,039 users over 100 days, we offer simple suggestions to designers of online communities to improve the quality of tags seen by their users.
© All rights reserved Sen et al. and/or ACM Press
Frankowski, Dan, Lam, Shyong K., Sen, Shilad, Harper, F. Maxwell, Yilek, Scott, Cassano, Michael and Riedl, John (2007): Recommenders everywhere: the WikiLens community-maintained recommender system. In: Proceedings of the 2007 International Symposium on Wikis 2007. pp. 47-60. Available online
Suppose you have a passion for items of a certain type, and you wish to start a recommender system around those items. You want a system like Amazon or Epinions, but for cookie recipes, local theater, or microbrew beer. How can you set up your recommender system without assembling complicated algorithms, large software infrastructure, a large community of contributors, or even a full catalog of items? WikiLens is open source software that enables anyone, anywhere to start a community-maintained recommender around any type of item. We introduce five principles for community-maintained recommenders that address the two key issues: (1) community contribution of items and associated information; and (2) finding items of interest. Since all recommender communities start small, we look at feasibility and utility in the small world, one with few users, few items, few ratings. We describe the features of WikiLens, which are based on our principles, and give lessons learned from two years of experience running wikilens.org.
© All rights reserved Frankowski et al. and/or ACM Press
Kapoor, Nishikant, Chen, Jilin, Butler, John T., Fouty, Gary C., Stemper, James A., Riedl, John and Konstan, Joseph A. (2007): TechLens: a researcher's desktop. In: Proceedings of the 2007 ACM Conference on Recommender Systems 2007. pp. 183-184. Available online
Rapid and continuous growth of digital libraries, coupled with brisk advancements in technology, has driven users to seek tools and services that are not only customized to their specific needs, but are also helpful in keeping them stay abreast with the latest developments in their field. TechLens is a recommender system that learns about its users through implicit feedback, builds correlations among them, and uses that information to generate recommendations that match the user's profile. It gives users control over which parts of their profile of known citations are used in forming recommendations for new articles. This demonstration is a prototype that showcases some of the tools and services that TechLens offers to the users of digital libraries.
© All rights reserved Kapoor et al. and/or ACM Press
Drenner, Sara, Harper, Max, Frankowski, Dan, Riedl, John and Terveen, Loren (2006): Insert movie reference here: a system to bridge conversation and item-oriented web sites. In: Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems 2006. pp. 951-954. Available online
Item-oriented Web sites maintain repositories of information about things such as books, games, or products. Many of these Web sites offer discussion forums. However, these forums are often disconnected from the rich data available in the item repositories. We describe a system, movie linking, that bridges a movie recommendation Web site and a movie-oriented discussion forum. Through automatic detection and an interactive component, the system recognizes references to movies in the forum and adds recommendation data to the forums and conversation threads to movie pages. An eight week observational study shows that the system was able to identify movie references with precision of .93 and recall of .78. Though users reported that the feature was useful, their behavior indicates that the feature was more successful at enriching the interface than at integrating the system.
© All rights reserved Drenner et al. and/or ACM Press
Rashid, Al Mamunur, Ling, Kimberly, Tassone, Regina D., Resnick, Paul, Kraut, Robert E. and Riedl, John (2006): Motivating participation by displaying the value of contribution. In: Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems 2006. pp. 955-958. Available online
One of the important challenges faced by designers of online communities is eliciting sufficient contributions from community members. Users in online communities may have difficulty either in finding opportunities to add value, or in understanding the value of their contributions to the community. Various social science theories suggest that showing users different perspectives on the value they add to the community will lead to differing amounts of contribution. The present study investigates a design augmentation for an existing community Web site that could benefit from additional contribution. The augmented interface includes individualized opportunities for contribution and an estimate of the value of each contribution to the community. The value is computed in one of four different ways: (1) value to self; (2) value to a small group the user has affinity with; (3) value to a small group the user does not have affinity with; and (4) value to the entire user community. The study compares the effectiveness of the different notions of value to 160 community members.
© All rights reserved Rashid et al. and/or ACM Press
Cosley, Dan, Frankowski, Dan, Terveen, Loren and Riedl, John (2006): Using intelligent task routing and contribution review to help communities build artifacts of lasting value. In: Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems 2006. pp. 1037-1046. Available online
Many online communities are emerging that, like Wikipedia, bring people together to build community-maintained artifacts of lasting value (CALVs). Motivating people to contribute is a key problem because the quantity and quality of contributions ultimately determine a CALV's value. We pose two related research questions: 1) How does intelligent task routing -- matching people with work -- affect the quantity of contributions? 2) How does reviewing contributions before accepting them affect the quality of contributions? A field experiment with 197 contributors shows that simple, intelligent task routing algorithms have large effects. We also model the effect of reviewing contributions on the value of CALVs. The model predicts, and experimental data shows, that value grows more slowly with review before acceptance. It also predicts, surprisingly, that a CALV will reach the same final value whether contributions are reviewed before or after they are made available to the community.
© All rights reserved Cosley et al. and/or ACM Press
Sen, Shilad, Lam, Shyong K., Rashid, Al Mamunur, Cosley, Dan, Frankowski, Dan, Osterhouse, Jeremy, Harper, F. Maxwell and Riedl, John (2006): tagging, communities, vocabulary, evolution. In: Proceedings of ACM CSCW06 Conference on Computer-Supported Cooperative Work 2006. pp. 181-190. Available online
A tagging community's vocabulary of tags forms the basis for social navigation and shared expression. We present a user-centric model of vocabulary evolution in tagging communities based on community influence and personal tendency. We evaluate our model in an emergent tagging system by introducing tagging features into the MovieLens recommender system. We explore four tag selection algorithms for displaying tags applied by other community members. We analyze the algorithms 'effect on vocabulary evolution, tag utility, tag adoption, and user satisfaction.
© All rights reserved Sen et al. and/or ACM Press
Cosley, Dan, Frankowski, Dan, Kiesler, Sara, Terveen, Loren and Riedl, John (2005): How oversight improves member-maintained communities. In: Proceedings of ACM CHI 2005 Conference on Human Factors in Computing Systems 2005. pp. 11-20. Available online
Online communities need regular maintenance activities such as moderation and data input, tasks that typically fall to community owners. Communities that allow all members to participate in maintenance tasks have the potential to be more robust and valuable. A key challenge in creating member-maintained communities is building interfaces, algorithms, and social structures that encourage people to provide high-quality contributions. We use Karau and Williams' collective effort model to predict how peer and expert editorial oversight affect members' contributions to a movie recommendation website and test these predictions in a field experiment with 87 contributors. Oversight increased both the quantity and quality of contributions while reducing antisocial behavior, and peers were as effective at oversight as experts. We draw design guidelines and suggest avenues for future work from our results.
© All rights reserved Cosley et al. and/or ACM Press
Resnick, Paul, Riedl, John, Terveen, Loren and Ackerman, Mark S. (2005): Beyond threaded conversation. In: Proceedings of ACM CHI 2005 Conference on Human Factors in Computing Systems 2005. pp. 2138-2139. Available online
Riedl, John, Jameson, Anthony, Billsus, Daniel and Lau, Tessa (eds.) Proceedings of the 10th international conference on Intelligent user interfaces January 10-13, 2005, San Diego, California, USA.
Amant, Robert St., Riedl, John and Jameson, Anthony (eds.) Proceedings of the 2005 International Conference on Intelligent User Interfaces January 10-13, 2005, San Diego, California, USA.
Herlocker, Jonathan L., Konstan, Joseph A., Terveen, Loren and Riedl, John (2004): Evaluating collaborative filtering recommender systems. In ACM Transactions on Information Systems, 22 (1) pp. 5-53. Available online
Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.
© All rights reserved Herlocker et al. and/or ACM Press
Miller, Bradley N., Konstan, Joseph A. and Riedl, John (2004): PocketLens: Toward a personal recommender system. In ACM Transactions on Information Systems, 22 (3) pp. 437-476. Available online
Recommender systems using collaborative filtering are a popular technique for reducing information overload and finding products to purchase. One limitation of current recommenders is that they are not portable. They can only run on large computers connected to the Internet. A second limitation is that they require the user to trust the owner of the recommender with personal preference data. Personal recommenders hold the promise of delivering high quality recommendations on palmtop computers, even when disconnected from the Internet. Further, they can protect the user's privacy by storing personal information locally, or by sharing it in encrypted form. In this article we present the new PocketLens collaborative filtering algorithm along with five peer-to-peer architectures for finding neighbors. We evaluate the architectures and algorithms in a series of offline experiments. These experiments show that Pocketlens can run on connected servers, on usually connected workstations, or on occasionally connected portable devices, and produce recommendations that are as good as the best published algorithms to date.
© All rights reserved Miller et al. and/or ACM Press
Lam, Shyong K. and Riedl, John (2004): Shilling recommender systems for fun and profit. In: Proceedings of the 2004 International Conference on the World Wide Web 2004. pp. 393-402. Available online
Recommender systems have emerged in the past several years as an effective way to help people cope with the problem of information overload. One application in which they have become particularly common is in e-commerce, where recommendation of items can often help a customer find what she is interested in and, therefore can help drive sales. Unscrupulous producers in the never-ending quest for market penetration may find it profitable to shill recommender systems by lying to the systems in order to have their products recommended more often than those of their competitors. This paper explores four open questions that may affect the effectiveness of such shilling attacks: which recommender algorithm is being used, whether the application is producing recommendations or predictions, how detectable the attacks are by the operator of the system, and what the properties are of the items being attacked. The questions are explored experimentally on a large data set of movie ratings. Taken together, the results of the paper suggest that new ways must be used to evaluate and detect shilling attacks on recommender systems.
© All rights reserved Lam and Riedl and/or ACM Press
Cosley, Dan, Lam, Shyong K., Albert, Istvan, Konstan, Joseph A. and Riedl, John (2003): Is seeing believing?: how recommender system interfaces affect users' opinions. In: Cockton, Gilbert and Korhonen, Panu (eds.) Proceedings of the ACM CHI 2003 Human Factors in Computing Systems Conference April 5-10, 2003, Ft. Lauderdale, Florida, USA. pp. 585-592.
Miller, Bradley N., Albert, Istvan, Lam, Shyong K., Konstan, Joseph A. and Riedl, John (2003): MovieLens unplugged: experiences with an occasionally connected recommender system. In: Johnson, Lewis and Andre, Elisabeth (eds.) International Conference on Intelligent User Interfaces 2003 January 12-15, 2003, Miami, Florida, USA. pp. 263-266. Available online
Recommender systems have changed the way people shop online. Recommender systems on wireless mobile devices may have the same impact on the way people shop in stores. We present our experience with implementing a recommender system on a PDA that is occasionally connected to the network. This interface helps users of the MovieLens movie recommendation service select movies to rent, buy, or see while away from their computer. The results of a nine month field study show that although there are several challenges to overcome, mobile recommender systems have the potential to provide value to their users today.
© All rights reserved Miller et al. and/or ACM Press
McNee, Sean, Lam, Shyong, Guetzlaff, Catherine, Konstan, Joseph A. and Riedl, John (2003): Confidence Displays and Training in Recommender Systems. In: Proceedings of IFIP INTERACT03: Human-Computer Interaction 2003, Zurich, Switzerland. p. 176.
Miller, B. N., Albert, I., Lam, S. K., Konstan, Joseph A. and Riedl, John (2003): MovieLens Unplugged: Experiences with a Recommender System on Four Mobile Devices. In: Proceedings of the HCI03 Conference on People and Computers XVII 2003. pp. 263-280.
McNee, Sean M., Lam, Shyong K., Konstan, Joseph A. and Riedl, John (2003): Interfaces for Eliciting New User Preferences in Recommender Systems. In: Brusilovsky, Peter, Corbett, Albert T. and Rosis, Fiorella De (eds.) User Modeling 2003 - 9th International Conference - UM 2003 June 22-26, 2003, Johnstown, PA, USA. pp. 178-187. Available online
McNee, Sean M., Albert, Istvan, Cosley, Dan, Gopalkrishnan, Prateep, Lam, Shyong K., Rashid, Al Mamunur, Konstan, Joseph A. and Riedl, John (2002): On the recommending of citations for research papers. In: Churchill, Elizabeth F., McCarthy, Joe, Neuwirth, Christine and Rodden, Tom (eds.) Proceedings of the 2002 ACM conference on Computer supported cooperative work November 16 - 20, 2002, New Orleans, Louisiana, USA. pp. 116-125. Available online
Collaborative filtering has proven to be valuable for recommending items in
many different domains. In this paper, we explore the use of collaborative
filtering to recommend research papers, using the citation web between papers
to create the ratings matrix. Specifically, we tested the ability of
collaborative filtering to recommend citations that would be suitable
additional references for a target research paper. We investigated six
algorithms for selecting citations, evaluating them through offline experiments
against a database of over 186,000 research papers contained in ResearchIndex.
We also performed an online experiment with over 120 users to gauge user
opinion of the effectiveness of the algorithms and of the utility of such
recommendations for common research tasks. We found large differences in the
accuracy of the algorithms in the offline experiment, especially when balanced
for coverage. In the online experiment, users felt they received quality
recommendations, and were enthusiastic about the idea of receiving
recommendations in this domain.
© All rights reserved McNee et al. and/or ACM Press
Rashid, Al Mamunur, Albert, Istvan, Cosley, Dan, Lam, Shyong K., McNee, Sean M., Konstan, Joseph A. and Riedl, John (2002): Getting to know you: learning new user preferences in recommender systems. In: Gil, Yolanda and Leake, David (eds.) International Conference on Intelligent User Interfaces 2002 January 13-16, 2002, San Francisco, California, USA. pp. 127-134. Available online
Recommender systems have become valuable resources for users seeking intelligent ways to search through the enormous volume of information available to them. One crucial unsolved problem for recommender systems is how best to learn about a new user. In this paper we study six techniques that collaborative filtering recommender systems can use to learn about new users. These techniques select a sequence of items for the collaborative filtering system to present to each new user for rating. The techniques include the use of information theory to select the items that will give the most value to the recommender system, aggregate statistics to select the items the user is most likely to have an opinion about, balanced techniques that seek to maximize the expected number of bits learned per presented item, and personalized techniques that predict which items a user will have an opinion about. We study the techniques thru offline experiments with a large pre-existing user data set, and thru a live experiment with over 300 users. We show that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions.
© All rights reserved Rashid et al. and/or ACM Press
Schafer, J. Ben, Konstan, Joseph A. and Riedl, John (2002): Meta-recommendation systems: user-controlled integration of diverse recommendations. In: Proceedings of the 2002 ACM CIKM International Conference on Information and Knowledge Management November 4-9, 2002, McLean, VA, USA. pp. 43-51. Available online
Connor, M. O., Cosley, Dan, Konstan, Joseph A. and Riedl, John (2001): PolyLens: A recommender system for groups of user. In: Ecscw 2001 - Proceedings of the Seventh European Conference on Computer Supported Cooperative Work 16-20 September, 2001, Bonn, Germany. pp. 199-218.
Riedl, John (2001): Guest Editor's Introduction: Personalization and Privacy. In IEEE Internet Computing, 5 (6) pp. 29-31. Available online
Herlocker, Jonathan L., Konstan, Joseph A. and Riedl, John (2000): Explaining Collaborative Filtering Recommendations. In: Kellogg, Wendy A. and Whittaker, Steve (eds.) Proceedings of the 2000 ACM conference on Computer supported cooperative work 2000, Philadelphia, Pennsylvania, United States. pp. 241-250. Available online
Automated collaborative filtering (ACF) systems predict a person's affinity for items or information by connecting that person's recorded interests with the recorded interests of a community of people and sharing ratings between like-minded persons. However, current recommender systems are black boxes, providing no transparency into the working of the recommendation. Explanations provide that transparency, exposing the reasoning and data behind a recommendation. In this paper, we address explanation interfaces for ACF systems - how they should be implemented and why they should be implemented. To explore how, we present a model for explanations based on the user's conceptual model of the recommendation process. We then present experimental results demonstrating what components of an explanation are the most compelling. To address why, we present experimental evidence that shows that providing explanations can improve the acceptance of ACF systems. We also describe some initial explorations into measuring how explanations can improve the filtering performance of users.
© All rights reserved Herlocker et al. and/or ACM Press
Herlocker, Jonathan L., Konstan, Joseph A., Borchers, Al and Riedl, John (1999): An Algorithmic Framework for Performing Collaborative Filtering. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1999. pp. 230-237. Available online
Sarwar, Badrul M., Konstan, Joseph A., Borchers, Al, Herlocker, Jonathan L., Miller, Brad and Riedl, John (1998): Using Filtering Agents to Improve Prediction Quality in the GroupLens Research Collaborative Filtering System. In: Poltrock, Steven and Grudin, Jonathan (eds.) Proceedings of the 1998 ACM conference on Computer supported cooperative work November 14 - 18, 1998, Seattle, Washington, United States. pp. 345-354. Available online
Collaborative filtering systems help address information overload by using the opinions of users in a community to make personal recommendations for documents to each user. Many collaborative filtering systems have few user opinions relative to the large number of documents available. This sparsity problem can reduce the utility of the filtering system by reducing the number of documents for which the system can make recommendations and adversely affecting the quality of recommendations. This paper defines and implements a model for integrating content-based ratings into a collaborative filtering system. The filterbot model allows collaborative filtering systems to address sparsity by tapping the strength of content filtering techniques. We identify and evaluate metrics for assessing the effectiveness of filterbots specifically, and filtering system enhancements in general. Finally, we experimentally validate the filterbot approach by showing that even simple filterbots such as spell checking can increase the utility for users of sparsely populated collaborative filtering systems.
© All rights reserved Sarwar et al. and/or ACM Press
Borchers, Al, Herlocker, Jonathan L. and Riedl, John (1998): Ganging up on Information Overload. In IEEE Computer, 31 (4) pp. 106-108.
Chi, Ed H. and Riedl, John (1998): An Operator Interaction Framework for Visualization Systems. In: InfoVis 1998 - IEEE Symposium on Information Visualization 19-20 October, 1998, Research Triangle Park, NC, USA. pp. 63-70. Available online
Chi, Ed Huai-hsin, Riedl, John, Barry, Phillip and Konstan, Joseph A. (1998): Principles for Information Visualization Spreadsheets. In IEEE Computer Graphics and Applications, 18 (4) pp. 30-38. Available online
Chi, Ed H., Konstan, Joseph A., Barry, Phillip and Riedl, John (1997): A Spreadsheet Approach to Information Visualization. In: Robertson, George G. and Schmandt, Chris (eds.) Proceedings of the 10th annual ACM symposium on User interface software and technology October 14 - 17, 1997, Banff, Alberta, Canada. pp. 79-80. Available online
In information visualization, as the volume and complexity of the data increases, researchers require more powerful visualization tools that allow them to more effectively explore multi-dimensional datasets. In this paper, we show a novel new visualization framework built upon the spreadsheet metaphor, where each cell can contain an entire dataset. Just as a numerical spreadsheet enables exploration of numbers, a visualization spreadsheet enables exploration of visualizations of data. Our prototype spreadsheets enabled users to compare visualizations in cells using the tabular layout. Users can use the spreadsheet to display, manipulate, and explore multiple visual representation techniques for their data. By applying different operations to the cells, we showed how visualization spreadsheets afford the construction of 'what-if' scenarios. The possible set of operations that users can apply consists of animation, filtering, and algebraic operators.
© All rights reserved Chi et al. and/or ACM Press
Chi, Ed H., Barry, Phillip, Riedl, John and Konstan, Joseph A. (1997): A spreadsheet approach to information visualization. In: InfoVis 1997 - IEEE Symposium on Information Visualization October 18-25, 1997, Phoenix, AZ, USA. pp. 17-24. Available online
Konstan, Joseph A., Miller, Bradley N., Maltz, David, Herlocker, Jonathan L., Gordon, Lee R. and Riedl, John (1997): GroupLens: Applying Collaborative Filtering to Usenet News. In Communications of the ACM, 40 (3) pp. 77-87.
Chi, Ed H., Riedl, John, Shoop, Elizabeth, Carlis, John V., Retzel, Ernest and Barry, Phillip (1996): Flexible Information Visualization of Multivariate Data from Biological Sequence Similarity Searches. In: IEEE Visualization 1996 1996. pp. 133-140.
Claypool, Mark and Riedl, John (1996): A Quality Planning Model for Distributed Multimedia in the Virtual Cockpit. In: ACM Multimedia 1996 1996. pp. 253-264.
Shoop, Elizabeth, Chi, Ed H., Carlis, John V., Bieganski, Paul, Riedl, John, Dalton, Neal, Newman, Thomas and Retzel, Ernest (1995): Implementation and testing of an automated EST processing and similarity analysis system. In: HICSS 1995 1995. pp. 52-61. Available online
Chi, Ed H., Barry, Phillip, Shoop, Elizabeth, Carlis, John V., Retzel, Ernest and Riedl, John (1995): Visualization of Biological Sequence Similarity Search Results. In: IEEE Visualization 1995 1995. pp. 44-51. Available online
Resnick, Paul, Iacovou, Neophytos, Suchak, Mitesh, Bergstrom, Peter and Riedl, John (1994): GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In: Proceedings of the 1994 ACM conference on Computer supported cooperative work October 22 - 26, 1994, Chapel Hill, North Carolina, United States. pp. 175-186. Available online
Collaborative filters help people make choices based on the opinions of other people. GroupLens is a system for collaborative filtering of netnews, to help people find articles they will like in the huge stream of available articles. News reader clients display predicted scores and make it easy for users to rate articles after they read them. Rating servers, called Better Bit Bureaus, gather and disseminate the ratings. The rating servers predict scores based on the heuristic that people who agreed in the past will probably agree again. Users can protect their privacy by entering ratings under pseudonyms, without reducing the effectiveness of the score prediction. The entire architecture is open: alternative software for news clients and Better Bit Bureaus can be developed independently and can interoperate with the components we have developed.
© All rights reserved Resnick et al. and/or ACM Press
Bieganski, Paul, Riedl, John, Carlis, John V. and Retzel, Ernest F. (1994): Generalized Suffix Trees for Biological Sequence Data: Applications and Implementation. In: HICSS 1994 1994. pp. 35-44.
Dewan, Prasun and Riedl, John (1993): Toward Computer-Supported Concurrent Software Engineering. In IEEE Computer, 26 (1) pp. 17-27.
Join our community and advance:
Page maintainer: The Editorial Team