research-article

Quality and Leniency in Online Collaborative Rating Systems

Authors:

Ke WangAuthors Info & Claims

ACM Transactions on the Web (TWEB), Volume 6, Issue 1

Article No.: 4, Pages 1 - 27

https://doi.org/10.1145/2109205.2109209

Published: 01 March 2012 Publication History

Abstract

The emerging trend of social information processing has resulted in Web users’ increased reliance on user-generated content contributed by others for information searching and decision making. Rating scores, a form of user-generated content contributed by reviewers in online rating systems, allow users to leverage others’ opinions in the evaluation of objects. In this article, we focus on the problem of summarizing the rating scores given to an object into an overall score that reflects the object’s quality. We observe that the existing approaches for summarizing scores largely ignores the effect of reviewers exercising different standards in assigning scores. Instead of treating all reviewers as equals, our approach models the leniency of reviewers, which refers to the tendency of a reviewer to assign higher scores than other coreviewers. Our approach is underlined by two insights: (1) The leniency of a reviewer depends not only on how the reviewer rates objects, but also on how other reviewers rate those objects and (2) The leniency of a reviewer and the quality of rated objects are mutually dependent. We develop the leniency-aware quality, or LQ model, which solves leniency and quality simultaneously. We introduce both an exact and a ranked solution to the model. Experiments on real-life and synthetic datasets show that LQ is more effective than comparable approaches. LQ is also shown to perform consistently better under different parameter settings.

References

[1]

Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6, 734--749.

Digital Library

[2]

Anton, H. and Rorres, C. 1987. Elementary Linear Algebra with Applications. John Wiley & Sons, Hoboken, NJ.

[3]

Arampatzis, A. and Kamps, J. 2009. A signal-to-noise approach to score normalization. In Proceeding of the 18th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 797--806.

Digital Library

[4]

Arkes, H. R. 2003. The nonuse of psychological research at two federal agencies. Psychol. Science 14, 1, 1--6.

[5]

Backstrom, L., Dwork, C., and Kleinberg, J. 2007. Wherefore art thou r3579x?: Anonymized social networks, hidden patterns, and structural steganography. In Proceedings of the 16th International World Wide Web Conference. ACM, New York, 181--190.

Digital Library

[6]

Bazerman, M. H. 1990. Judgment in Managerial Decision-Making 2nd Ed. Wiley, Hoboken, NJ.

[7]

Bell, R. M. and Koren, Y. 2007. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the 7th IEEE International Conference on Data Mining. 43--52.

Digital Library

[8]

Bhattacharjee, R. and Goel, A. 2005. Avoiding ballot stuffing in Ebay-like reputation systems. In Proceeding of the ACM SIGCOMM Workshop on Economics of Peer-to-Peer Systems. ACM, New York, 133--137.

Digital Library

[9]

Blackburn, J. L. and Hakel, M. D. 2006. An examination of sources of peer-review bias. Psychol. Science 17, 5, 378--382.

[10]

Borgatti, S. P. and Everett, M. G. 1997. Network analysis of 2-mode data. Social Netw. 19, 3, 243--269.

[11]

Borodin, A., Roberts, G. O., Rosenthal, J. S., and Tsaparas, P. 2005. Link analysis ranking: Algorithms, theory, and experiments. ACM Trans. Internet Technol. 5, 1, 231--297.

Digital Library

[12]

Busenitz, L. and Lau, C. 1996. A cross-cultural cognitive model of new venture creation. Entrepreneurship: Theory Pract. 20, 4, 25--39.

[13]

Chen, M. and Singh, J. P. 2001. Computing and using reputations for internet ratings. In Proceedings of the 3rd ACM Conference on Electronic Commerce. ACM, New York, 154--162.

Digital Library

[14]

Dellarocas, C. 2000. Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior. In Proceedings of the 2nd ACM Conference on Electronic Commerce. ACM, New York, 150--157.

Digital Library

[15]

Dumais, S. and Nielsen, J. 1992. Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 233--244.

Digital Library

[16]

Dwork, C., Kumar, R., Naor, M., and Sivakumar, D. 2001. Rank aggregation methods for the Web. In Proceedings of the 10th International World Wide Web Conference. 613--622.

Digital Library

[17]

Fagin, R., Kumar, R., and Sivakumar, D. 2003. Comparing top k lists. SIAM J. Discrete Math. 17, 1, 134--160.

Digital Library

[18]

Faust, K. 1997. Centrality in affiliation networks. Social Netw. 19, 2, 157--191.

[19]

Fernandez, M., Vallet, D., and Castells, P. 2006. Probabilistic score normalization for rank aggregation. In Proceedings of the European Conference on Information Retrieval. 553--556.

Digital Library

[20]

Figueira, J., Greco, S., and Ehrgott, M., Eds. 2005. Multiple Criteria Decision Analysis: State of the Art Surveys. Springer Science and Business Media, Inc.

[21]

Geller, J. and Scherl, R. 1997. Challenge: Technology for automated reviewer selection. In Proceedings of the International Joint Conferences on Artificial Intelligence. 55--61.

[22]

Golbeck, J. and Hendler, J. 2006. Inferring binary trust relationships in Web-based social networks. ACM Trans. Internet Technol. 6, 4, 497--529.

Digital Library

[23]

Golub, G. H. and Van Loan, C. F. 1996. Matrix Computations 3rd Ed. Johns Hopkins University Press.

Digital Library

[24]

Grimmett, G. R. and Stirzaker, D. R. 1982. Probability and Random Processes. Oxford University Press, Oxford, UK.

[25]

Guha, R., Kumar, R., Raghavan, P., and Tomkins, A. 2004. Propagation of trust and distrust. In Proceedings of the International World Wide Web Conference. ACM, New York.

Digital Library

[26]

Haveliwala, T. H. 2003. Topic-sensitive PageRank: A context-sensitive ranking algorithm for Web search. IEEE Trans. Knowl. Data Engin. 15, 4, 784--796.

Digital Library

[27]

Herlocker, J. L., Konstan, J. A., and Riedl, J. 2000. Explaining collaborative filtering recommendations. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM, New York, NY, 241--250.

Digital Library

[28]

Herlocker, J., Konstan, J. A., and Riedl, J. 2002. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Information Retrieval 5, 287--310.

Digital Library

[29]

Hettich, S. and Pazzani, M. J. 2006. Mining for proposal reviewers: Lessons learned at the national science foundation. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, 862--871.

Digital Library

[30]

Jin, R. and Si, L. 2004. A study of methods for normalizing user ratings in collaborative filtering. In Proceedings of the 27th ACM SIGIR Conference on Research and Development in Information Retrieval. 568--569.

Digital Library

[31]

Jin, R., Si, L., Zhai, C.-X., and Callan, J. 2003. Collaborative filtering with decoupled models for preferences and ratings. In Proceedings of the International Conference on Information and Knowledge Management. ACM, New York, NY, 309--316.

Digital Library

[32]

Jindal, N. and Liu, B. 2007. Analyzing and detecting review spam. In Proceedings of the IEEE International Conference on Data Mining. IEEE Computer Society, Los, Alamitos, CA.

Digital Library

[33]

Kleinberg, J. M. 1999. Authoritative sources in a hyperlinked environment. J. ACM 46, 5, 604--632.

Digital Library

[34]

Korhonen, P., Moskowitz, H., and Wallenius, J. 1992. Multiple criteria decision support---a review. Euro. J. Operational Res. 63, 3, 361--375.

[35]

Lam, S. K. and Riedl, J. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th International World Wide Web Conference. ACM, New York, 393--402.

Digital Library

[36]

Lauw, H. W., Lim, E.-P., and Wang, K. 2007. Summarizing review scores of “unequal” reviewers. In Proceedings of the 2007 SIAM International Conference on Data Mining.

[37]

Lemire, D. 2005. Scale and translation invariant collaborative filtering systems. Info. Retrieval 8, 1, 129--150.

Digital Library

[38]

Ma, H., Yang, H., Lyu, M., and King, I. 2008. Sorec: social recommendation using probabilistic matrix factorization. In CIKM. ACM, New York, NY, 931--940.

Digital Library

[39]

Ma, H., King, I., and Lyu, M. R. 2009. Learning to recommend with social trust ensemble. In SIGIR. ACM, New York, NY, 203--210.

Digital Library

[40]

Manmatha, R. and Sever, H. 2002. A formal approach to score normalization for meta-search. In Proceedings of the 2nd International Conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc., San Francisco, CA, 98--103.

Digital Library

[41]

Massa, P. and Avesani, P. 2005. Controversial users demand local trust metrics: An experimental study on epinions.com community. In AAAI. AAAI Press, 121--126.

Digital Library

[42]

Mobasher, B., Burke, R., and Sandvig, J. 2006. Model-based collaborative filtering as a defense against profile injection attacks. In Proceedings of the National Conference on Artificial Intelligence.

Digital Library

[43]

Page, L., Brin, S., Motwani, R., and Winograd, T. 1998. The PageRank citation ranking: Bringing order to the Web. Stanford Digital Library Technologies Project.

[44]

Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 175--186.

Digital Library

[45]

Riggs, T. and Wilensky, R. 2001. An algorithm for automated rating of reviewers. In Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries. ACM, New York, 381--387.

Digital Library

[46]

Sarwar, B. M., Karypis, G., Konstan, J. A., and Riedl, J. 2000. Application of dimensionality reduction in recommender system -- a case study. In Proceedings of the ACM WebKDD Web Mining for E-Commerce Workshop.

[47]

Shen, J., Lin, Y., Xue, G.-R., Zhu, F.-D., and Yao, A.-G. 2006. IRFCF: Iterative rating filling collaborative filtering algorithm. In Proceedings of the 8th Asia-Pacific Web Conference. 862--867.

Digital Library

[48]

Simon, M., Houghton, S. M., and Aquino, K. 1999. Cognitive biases, risk perception, and venture formation: How individuals decide to start companies. J. Bus. Venturing 15, 113--134.

[49]

Tantipathananandh, C., Berger-Wolf, T., and Kempe, D. 2007. A framework for community identification in dynamic social networks. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, 717--726.

Digital Library

[50]

Traupman, J. and Wilensky, R. 2004a. Collaborative quality filtering: Establishing consensus or recovering ground truth? In Proceedings of the KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Digital Library

[51]

Traupman, J. and Wilensky, R. 2004b. Collaborative quality filtering: Establishing consensus or recovering ground truth? In Proceedings of the KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 73--86.

Digital Library

[52]

Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. 2002. Probability & Statistics for Engineers & Scientists 7th Ed. Prentice Hall, NJ.

[53]

Wasserman, S. and Faust, K. 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, Cambridge, UK.

[54]

Yang, B., Cheung, W., and Liu, J. 2007. Community mining from signed social networks. IEEE Trans. Knowl. Data Engin. 19, 10, 1333--1348.

Digital Library

[55]

Zhang, J. and Cohen, R. 2006. Trusting advice from other buyers in e-marketplaces: The problem of unfair ratings. In Proceedings of the 8th International Conference on Electronic Commerce: The new e-Commerce: Innovations for Conquering Current Barriers, Obstacles and Limitations to Conducting Successful Business on the Internet. ACM, New York, 225--234.

Digital Library

[56]

Zhou, D., Manavoglu, E., Li, J., Giles, C. L., and Zha, H. 2006. Probabilistic models for discovering e-communities. In Proceedings of the 15th International World Wide Web Conference. ACM, New York, 173--182.

Digital Library

Cited By

Hung CLim E(2021)On Aggregating Salaries of Occupations From Job Post and Review DataIEEE Access10.1109/ACCESS.2021.30662049(43422-43433)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3066204
Alqwadri AAzzeh MAlmasalha F(2021)Application of Machine Learning for Online Reputation SystemsInternational Journal of Automation and Computing10.1007/s11633-020-1275-7Online publication date: 8-Mar-2021
https://doi.org/10.1007/s11633-020-1275-7
Koncar PHelic D(2020)Employee Satisfaction in Online ReviewsSocial Informatics10.1007/978-3-030-60975-7_12(152-167)Online publication date: 6-Oct-2020
https://dl.acm.org/doi/10.1007/978-3-030-60975-7_12
Show More Cited By

Index Terms

Quality and Leniency in Online Collaborative Rating Systems
1. Applied computing
  1. Law, social and behavioral sciences
2. Information systems
  1. Information systems applications

Recommendations

A Rating-Ranking Method for Crowdsourced Top-k Computation
SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

Crowdsourced top- k computation aims to utilize the human ability to identify Top- k objects from a given set of objects. Most of existing studies employ a pairwise comparison based method, which first asks workers to compare each pair of objects and ...
A document rating system for preference judgements
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

High quality relevance judgments are essential for the evaluation of information retrieval systems. Traditional methods of collecting relevance judgments are based on collecting binary or graded nominal judgments, but such judgments are limited by ...
Computing and using reputations for internet ratings
EC '01: Proceedings of the 3rd ACM conference on Electronic Commerce

Ratings for products and services are increasingly important on the Internet, as they allow users to harvest the wisdom of the community in making decisions. However, the difficulty with ratings is that little is known about the people providing them. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on the Web

ACM Transactions on the Web Volume 6, Issue 1

March 2012

109 pages

ISSN:1559-1131

EISSN:1559-114X

DOI:10.1145/2109205

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 March 2012

Accepted: 01 October 2011

Revised: 01 March 2011

Received: 01 March 2009

Published in TWEB Volume 6, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
593
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hung CLim E(2021)On Aggregating Salaries of Occupations From Job Post and Review DataIEEE Access10.1109/ACCESS.2021.30662049(43422-43433)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3066204
Alqwadri AAzzeh MAlmasalha F(2021)Application of Machine Learning for Online Reputation SystemsInternational Journal of Automation and Computing10.1007/s11633-020-1275-7Online publication date: 8-Mar-2021
https://doi.org/10.1007/s11633-020-1275-7
Koncar PHelic D(2020)Employee Satisfaction in Online ReviewsSocial Informatics10.1007/978-3-030-60975-7_12(152-167)Online publication date: 6-Oct-2020
https://dl.acm.org/doi/10.1007/978-3-030-60975-7_12
Braga DNiemann MHellingrath BNeto F(2018)Survey on Computational Trust and Reputation ModelsACM Computing Surveys10.1145/323600851:5(1-40)Online publication date: 19-Nov-2018
https://dl.acm.org/doi/10.1145/3236008
Xie HLui J(2018)Incentive Mechanism and Rating System Design for Crowdsourcing Systems: Analysis, Tradeoffs and InferenceIEEE Transactions on Services Computing10.1109/TSC.2016.253995411:1(90-102)Online publication date: 1-Jan-2018
https://doi.org/10.1109/TSC.2016.2539954
Ivanova OScholz M(2017)How can online marketplaces reduce rating manipulation? A new approach on dynamic aggregation of online ratingsDecision Support Systems10.1016/j.dss.2017.10.003104(64-78)Online publication date: Dec-2017
https://doi.org/10.1016/j.dss.2017.10.003
Shaalan YZhang X(2016)A Time and Opinion Quality-Weighted Model for Aggregating Online ReviewsDatabases Theory and Applications10.1007/978-3-319-46922-5_21(269-282)Online publication date: 21-Sep-2016
https://doi.org/10.1007/978-3-319-46922-5_21
Xie HLui J(2015)Mathematical Modeling and Analysis of Product Rating with Partial InformationACM Transactions on Knowledge Discovery from Data10.1145/27003869:4(1-33)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1145/2700386
Abdel-Hafez AXu YJosang A(2015)An accurate rating aggregation method for generating item reputation2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA)10.1109/DSAA.2015.7344804(1-8)Online publication date: Oct-2015
https://doi.org/10.1109/DSAA.2015.7344804
Zhang FZhou Q(2015)Ensemble detection model for profile injection attacks in collaborative recommender systems based on BP neural networkIET Information Security10.1049/iet-ifs.2013.01459:1(24-31)Online publication date: 1-Jan-2015
https://doi.org/10.1049/iet-ifs.2013.0145
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents