Abstract
Recent studies have indicated that the application of Multi-Criteria Decision Making (MCDM) methods in recommender systems has yet to be systematically explored. This observation partially contradicts with the fact that in related literature, there exist several contributions describing recommender systems that engage some MCDM method. Such systems, which we refer to as multi-criteria recommender systems, have early demonstrated the potential of applying MCDM methods to facilitate recommendation, in numerous application domains. On the other hand, a comprehensive analysis of existing systems would facilitate their understanding and development. Towards this direction, this paper identifies a set of dimensions that distinguish, describe and categorize multi-criteria recommender systems, based on existing taxonomies and categorizations. These dimensions are integrated into an overall framework that is used for the analysis and classification of a sample of existing multi-criteria recommender systems. The results provide a comprehensive overview of the ways current multi-criteria recommender systems support the decision of online users.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Adomavicius, G., Sankaranarayanan, R., Sen, S., Tuzhilin, A.: Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans. Inf. Syst. 23(1), 103–145 (2005)
Adomavicius, G., Tuzhilin, A.: Multidimensional recommender systems: a data warehousing approach. In Proc. 2nd Int. Worksh. Electr. Comm. (WELCOM’01), LCCS vol. 2232, pp. 180–192. Springer, Berlin Heidelberg New York (2001)
Adomavicius, G., Tuzhilin, A.: Towards the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)
Ardissono, L., Goy, A., Petrone, G., Segnan, M., Torasso, P.: Intrigue: personalised recommendation of tourist attractions for desktop and handset devices. In: Applied Artificial Intelligence Special Issue on ‘Artificial Intelligence for Cultural Heritage and Digital Libraries’ vol. 17(8–9), pp. 687–714. Taylor & Thomas, New York (2003)
Ariely, D., Lynch, J.G.,Jr., Aparicio, M.: Learning by collaborative and individual-based recommendation agents. J. Consum. Psych. 14(1&2), 81–94 (2004)
Balabanovic, M., Shoham, Y.: Fab: content-based, collaborative recommendation. Comm. ACM 40(3), 66–72 (1997)
Baudisch, P.: Dynamic information filtering. PhD thesis, Darmstad Technical University, GMD Research Series No. 16 (2001)
Belkin, N.J.: Helping people find what they don’t know. Comm. ACM 43(8), 58–61 (2000)
Belkin, N.J., Croft, W.B.: Information filtering and information retrieval: two sides of the same coin? Comm. ACM 35(12), 29–38 (1992)
Bharati, P., Chaudhury, A.: An empirical investigation of decision-making satisfaction in web-based decision support systems. Dec. Supp. Syst. 1051, 1–11 (2003)
Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In Proc. 14th Conf. Uncert. Art. Intell., Madison WI, USA, July. Morgan Kaufmann, San Francisco (1998)
Brusilovsky, P.: Methods and techniques of adaptive hypermedia. User Model. User Adapt. Inter. 6(2–3), 87–129 (1996)
Brusilovsky, P., Vassileva, J.: Course sequencing techniques for large-scale web based education. Int. J. Contin. Eng. Educ. Lifelong Learn. 13(1/2), 75–94 (2003)
Burke, R.: Hybrid recommender systems: survey and experiments. User Model. User Adapt. Inter. 12, 331–370 (2002)
Carenini, G.: User-specific decision-theoretic accuracy metrics for collaborative filtering. In Proc. ‘Beyond Personalization’ Worksh., Intell. User Interf. Conf., San Diego, California, USA (2005)
Changchien, S.W., Lu, T.-C.: Mining association rules procedure to support on-line recommendation by customers and products fragmentation. Exp. Syst. Appls. 20, 325–335 (2001)
Cheetham, W.: Global grade selector: a recommender system for supporting the sale of plastic resin. Tech. Inf. Series, GE Global Research, TR. 2003GRC261, December. Springer, Berlin Heidelberg New York (2003)
Cho, Y.H., Kim, J.K.: Application of web usage mining and product taxonomy to collaborative recommendations in e-commerce. Exp. Syst. Appl. 26, 233–246 (2004)
Cho, Y.H., Kim, J.K., Kim, S.H.: A personalised recommender system based on web usage mining and decision tree induction. Exp. Syst. Appl. 23, 329–342 (2002)
Choi, S.H., Cho, Y.H.: An utility range-based similar product recommendation algorithm for collaborative companies. Exp. Syst. Appl. 27, 549–557 (2004)
Claypool, M., Gokhale, A., Miranda, T.: Combining content-based and collaborative filters in an online newspaper. In Proc. ACM SIGIR Worksh. Recomm. Syst. Impl. Eval., Berkeley CA, USA (1999)
Cohen, W.W., Fan, W.: Web-collaborative filtering: recommending music by crawling the web. In Proc. 9th Int. WWW Conf., Amsterdam, Netherlands. North-Holland (2000)
Cunliffe, D.: Developing usable web sites—a review and model. Int. Res.: El. Netw. Appl. Policy 10(4), 295–307 (2000)
De Bra, P., Houben, G.-J., Wu, H.: AHAM: a Dexter-based reference model for adaptive hypermedia. In Proc. 10th Conf. Hypertext Hypermedia, Darmstadt, Germany, pp. 147–156. ACM, New York (1999)
Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2004)
DIMACS and LAMSADE: Computer science and decision theory: applications of notions of consesus. DIMACS/LAMSADE Partnership (http://dimacs.rutgers.edu/Workshops/Lamsade/) (2004)
Falle, W., Stoefler, D., Russ, C., Zanker, M., Felfernig, A.: Using knowledge-based advisor technology for improved customer satisfaction in the shoe industry. In Proc. Int. Conf. Econ. Techn. Organis. Asp. of Product Config. Syst., Technical University of Denmark, Kopenhagen (2004)
Ghosh, S., Munde, M., Hernandez, K., Sen, S.: Voting for movies: the anatomy of a recommender system. In Proc. Aut. Agents, Seattle WA USA. ACM, New York (1999)
Goldberg, D., Nichols, D., Oki, B.M., Terry, D.: Using collaborative filtering to weave an information tapestry. Comm. ACM 35(12), 61–70 (1992)
Goldberg, K., Roeder, T., Gupta, D., Perkins, C.: Eigentaste: a constant time collaborative filtering algorithm. Inf. Retr. J. 4(2), 133–151 (2001)
Guan, S., Ngoo, C.S., Zhu, F.: Handy broker: an intelligent product-brokering agent for m-commerce applications with user preference tracking. Electr. Comm. Res. App. 1, 314–330 (2002)
Ha, V., Haddawy, P.: Similarity of personal preferences: theoretical foundations and empirical analysis. Artif. Intell. 146(2), 149–173 (2003)
Han, P., Xie, B., Yang, F., Shen, R.: A scalable P2P recommender system based on distributed collaborative filtering. Exp. Syst. Appl. 27, 203–210 (2004)
Hanani, U., Shapira, B., Shoval, P.: Information filtering: overview of issues, research and systems. User Model. User Adapt. Inter. 11, 203–259 (2001)
Herlocker, J., Konstan, J.A., Riedl, J.: An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Inf. Retr. 5, 287–310 (2002)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)
Herrera-Viedma, E., Pasi, G., Lopez-Herrera, A.G.: Evaluating the information quality of web sites: a qualitative methodology based on fuzzy computing with words. Res. Group Soft Comp. Intell. Inf. Syst., Univ. of Granada, TR. #SCI2S-2004-10 (2004)
Huang, Z., Chen, H., Zeng, D.: Applying associative retrieval techniques to alleviate the sparsity problem in collaborative filtering. ACM Trans. Inf. Syst. 22(1), 116–142, January (2004)
Hung, L.-P.: A personalized recommendation system based on product taxonomy for one-to-one marketing online. Exp. Syst. Appl. 29, 383–392 (2005)
Jacquet-Lagreze, E., Siskos, J.: Assessing a set of additive utility functions for multicriteria decision-making: the UTA method. Eur. J. Oper. Res. 10, 151–164 (1982)
Jacquet-Lagreze, E., Siskos, Y.: Preference disaggregation: 20 years of MCDA experience. Eur. J. Oper. Res. 130, 233–245 (2001)
Karacapilidis, N., Hatzieleutheriou, L.: A hybrid framework for similarity-based recommendations. Int. J. Bus. Intell. Data Min. 1(1) (2005)
Karampiperis, P., Sampson, D.: Adaptive learning resources sequencing in educational hypermedia systems. Educ. Techn. Soc. 8(4), 128–147 (2005)
Keeney, R.L.: Value-focused Thinking: A Path to Creative Decisionmaking. Harvard University Press, Cambridge, MA (1992)
Kerschberg, L., Kim, W., Scime, A.: WebSifter II: A Personalizable Meta-Search Agent Based on Semantic Weighted Taxonomy Tree. In Proc. Int. Conf. Internet Comp., Las Vegas, Nevada, USA (2001)
Kim, W., Kerschberg, L., Scime, A.: Learning for automatic personalization in a semantic taxonomy-based meta-search agent. Electr. Comm. Res. App. 1, 150–173 (2002)
Kim, Y.S., Yum, B.-J., Kim, S.M.: Development of a recommender system based on navigational and behavioral patterns of customers in e-commerce sites. Exp. Syst. Appl. 28, 381–393 (2005)
Kleinberg, J., Sandler, M.: Convergent Algorithms for Collaborative Filtering. In Proc. ACM E-Comm. Conf., San Diego California, USA. ACM, New York (2003)
Konstan, J.A.: Introduction to recommender systems: algorithms and evaluation. ACM Trans. Inf. Syst. 22(1), 1–4 (2004)
Konstan, J.A., Miller, B.N., Maltz, D., Herlocker, J.L., Gordon, L.R., Riedl, J.: Grouplens: applying collaborative filtering to usenet news. Comm. ACM 40(3), 77–87, March (1997)
Kwon, O.B.: I know what you need to buy: context-aware multimedia-based recommendation system. Exp. Syst. Appl. 25, 387–400 (2003)
Lee, W.-P.: Towards agent-based decision making in the electronic marketplace: interactive recommendation and automated negotiation. Exp. Syst. Appl. 27, 665–679 (2004)
Lee, W.-P., Yang, T.-H.: Personalizing information appliances: a multi-agent framework for TV programme recommendations. Exp. Syst. Appl. 25, 331–341 (2003)
Lee, W.-P., Liu, C.-H., Lu, C.-C.: Intelligent agent-based systems for personalized recommendations in internet commerce. Exp. Syst. Appl. 22, 275–284 (2002)
Lee, J.-S., Jun, C.-H., Lee, J., Kim, S.: Classification-based collaborative filtering using market basket data. Exp. Syst. Appl. 29, 700–704 (2005)
Li, Y., Lu, L., Xuefeng, L.: A hybrid collaborative filtering method for multiple-interests and multiple-content recommendation in E-Commerce. Exp. Syst. Appl. 28, 67–77 (2005)
Lihua, W., Lu, L., Jing, L., Zongyong, L.: Modeling multiple interests by an improved GCS approach. Exp. Syst. Appl. 29(4), 757–767 (2005)
Liu, D.-R., Shih, Y.-Y.: Integrating AHP and data mining for product recommendation based on customer lifetime value. Inf. Manag. 42, 387–400 (2005)
Malone, T., Grant, K., Turbak, F., Brobst, S., Cohen, M.: Intelligent information sharing systems. Comm. ACM 30(5), 390–402 (1987)
Manouselis, N., Costopoulou, C.: Designing a web-based testing tool for multi-criteria recommender systems. Eng. Lett., Sp. Iss. “Web Engineering,” in press
Manouselis, N., Costopoulou, C.: Experimental analysis of design choices in a multi-attribute utility collaborative filtering system. Int. J. Patt. Recogn. Art. Int., Sp. Iss. “Personalization Techniques for Recommender Systems and Intelligent User Interfaces”, in press
Manouselis, N., Costopoulou, C.: Quality in metadata: a schema for E-commerce. Online Inf. Review, Sp. Iss. “Advances in Digital Information Services and Metadata Research” 30(3), 217–237 (2006)
Manouselis, N., Costopoulou, C.: Towards a design process for intelligent product recommendation services in e-markets. In: Zha, X. (ed.) Artif. Intell. Integr. Inf. Syst.: Emerg. Techn. App., Hershey, PA: Idea Group Publishing, 398–417 (2007)
Manouselis, N., Sampson, D.: A multi-criteria model to support automatic recommendation of e-learning quality approaches. In Proc. 16th World Conf. Educ. Multim. Hyperm. Telecomm., Lugano, Switzerland. AACE (2004)
Martin-Guerrero, J.D., Palomares, A., Balaguer-Ballester, E., Soria-Olivas, E., Gomez-Sanchis, J., Soriano-Asensi, A.: Studying the feasibility of a recommender in a citizen web portal based on user modeling and clustering algorithms. Exp. Syst. Appl. 30(2), 299–312 (2006)
Middleton, S.E., Shadbolt, N.R., Roure, D.C.: Ontological user profiling in recommender systems. ACM Trans. Inf. Syst. 22(1), 54–88 (2004)
Miller, B.N., Konstan, J.A., Riedl, J.: PocketLens: toward a personal recommender system. ACM Trans. Inf. Syst. 22(3), 437–476 (2004)
Min, S.-H., Han, I.: Detection of the customer time-invariant pattern for improving recommender systems. Exp. Syst. Appl. 28, 189–199 (2005)
Minio, M., Tasso, C.: User modeling for information filtering on internet services: exploiting an extended version of the UMT shell. In Proc. Work. User Mod. Inf. Filt. WWW, User Mod. Conference (UM‘96), Kailua-Koua, Hawaii (1996)
Mirza, B.J., Keller, B.J., Ramakrishnan, N.: Studying recommendation algorithms by graph analysis. J. Intell. Inf. Syst. 20(2), 131–160, March (2003)
Montaner, M., Lopez, B., de la Rosa, J.L.: A taxonomy of recommender agents on the internet. Artif. Intell. Rev. 19, 285–330 (2003)
Montaner, M., Lopez, B., de la Rosa, J.L.: Opinion-based filtering through trust. In Proc. 6th Int. Worksh. Cooper. Inf. Agents VI, LNCS vol. 2446, pp. 164–178, Springer, Berlin Heidelberg New York (2002)
Mukherjee, R., Dutta, P.S., Jonsdottir, G., Sen, S.: MOVIES2GO—an online voting based movie recommender system. In Proc. ACM AGENTS, Montreal Quebec Canada, May. ACM, New York (2001)
Nguyen, H., Haddawy, P.: DIVA: applying decision theory to collaborative filtering. In Proc. AAAI Worksh. Recomm. Syst., Madison, WI, July (1998)
Nguyen, H., Haddawy, P.: The decision-theoretic video advisor. In Proc. 15th Conf. Uncert. Artif. Intell., Stockholm, Sweden, pp. 494–501. AAAI, Madison, WI (1999)
Noh, S.: Implementing purchasing assistant using personal profile. In Proc. IADIS Int. Conf. App. Comp., Lisbon, Portugal, March. Kluwer, The Netherlands (2004)
Pazzani, M., Billsus, D.: Learning and revising user profiles: the identification of interesting web sites. Mach. Learn. 27, 313–331 (1997)
Pennock, D.M., Horvitz, E.: Analysis of the axiomatic foundations of collaborative filtering. In Proc. AAAI Worksh. Artif. Intell. Electr. Comm., Orlando, Florida, July. AAAI (1999)
Perny, P., Zucker, J.-D.: Collaborative filtering methods based on fuzzy preference relations. In Proc. EUROFUSE-SIC, pp. 279–285. Kluwer, The Netherlands (1999)
Perny, P., Zucker, J.-D.: Preference-based search and machine learning for collaborative filtering: the ‘film-conseil’ movie recommender system. Inform. Interact. Intell. 1(1), 9–48 (2001)
Perugini, S., Goncalves, M.A., Fox, E.A.: Recommender systems research: a connection-centric survey. J. Intell. Inf. Syst. 23(2), 107–143, September (2004)
Plantie, M., Montmain, J., Dray, G.: Movies recommenders systems: automation of the information and evaluation phases in a multi-criteria decision-making process. In: Andersen, K.V., Debenham, J., Wagner, R. (eds.) Proc. DEXA, LNCS vol. 3588, pp. 633–644. Springer, Berlin Heidelberg New York (2005)
Price, B., Messinger, P.R.: Optimal recommendation sets: covering uncertainty over user preferences. Informs Ann. Meet., Denver 2004, AAAI (2005)
Reich, E.: Users are individuals: individualizing user models. Int. J. Man-Mach. Stud. 18, 199–214 (1983)
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: GroupLens: an open architecture for collaborative filtering. In Proc. ACM CSCW, pp. 175–186. ACM, New York (1994)
Resnick, P., Varian, H.R.: Recommender systems. Comm. ACM 40(3), 56–58 (1997)
Roh, T.H., Oh, K.J., Han, I.: The collaborative filtering recommendation based on SOM cluster-indexing CBR. Exp. Syst. Appl. 25, 413–423 (2003)
Roy, B., Bouyssou, D.: Aide Multicritere a La Decision: Methodes et Cas. Economica, Paris (1993)
Roy, B.: Multicriteria Methodology for Decision Aiding. Kluwer, Dordrecht (1996)
Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Analysis of Recommendation Algorithms for E-Commerce. In Proc. of the ACM El. Comm. (EC’00), Minneapolis, Minnesota. Kluwer, The Netherlands (2000)
Schaefer, R.: Rules for Using Multi-Attribute Utility Theory for Estimating a User’s Interests. In Proc. ABIS Worksh. ‘Adaptivität und Benutzermodellierung in interaktiven Softwaresystemen’, Dortmund, Germany, October (2001)
Schafer, J.B.: DynamicLens: a dynamic user-interface for a meta-recommendation system. In Proc. “Beyond Personalization 2005” Work., Int. Conf. Intell. User Interf. (IUI 2005), San Diego, CA, January 9. Information Science (2005)
Schafer, J.B., Konstan, J.A., Riedl, J.: E-commerce recommendation applications. Data Min. Knowl. Discov. 5, 115–153 (2001)
Schein, A.I., Popescul, A., Ungar, L., Pennock, D.M.: CROC: a new evaluation criterion for recommender systems. Electr. Comm. Res. 5, 51–74 (2005)
Schickel-Zuber, V., Faltings, B.: Hetereogeneous attribute utility model: a new approach for modelling user profiles for recommendation systems. In Proc. Worksh. Knowl. Discov. in the Web, Chicago, Illinois, USA, August (2005)
Schmitt, C., Dengler, D., Bauer, M.: The MAUT-Machine: An Adaptive Recommender System. In Proc. ABIS Worksh. ‘Adaptivität und Benutzermodellierung in interaktiven Softwaresystemen,’ Hannover, Germany, October (2002)
Shardanand, U., Maes, P.: Social Information Filtering: Algorithms for Automatic ‘Word of Mouth.’ In Proc. Conf. Hum. Fact. Comput. Syst., Denver CO, USA. Addison-Wesley, New York (1995)
Smyth, B., Cotter, P.: A personalized television listings service. Comm. ACM 43(8), 107–111, August (2000)
Srikumar, K., Bhasker, B.: Personalized product selection in internet business. J. Electr. Comm. Res. 5(4), 216–227 (2004)
Stolze, M., Rjaibi, W.: Towards scalable scoring for preference-based item recommendation. IEEE Data Engin. Bullet. 24(3), 42–49 (2001)
Stolze, M., Stroebel, M.: Dealing with Learning in eCommerce Product Navigation and Decision Support: The Teaching Salesman Problem. In Proc. 2nd World Congr. Mass Custom. Person., Munich, Germany (2003)
Terveen, L., Hill, W., Amento, B., McDonald, D., Creter, J.: PHOAKS: a system for sharing recommendations. Comm. ACM 40(3), 59–65, March (1997)
Tewari, G., Youll, J., Maes, P.: Personalized location-based brokering using an agent-based intermediary architecture. Dec. Supp. Syst. 34, 127–137 (2002)
Tso, K.H.L., Schmidt-Thieme, L.: Evaluation of attribute-aware recommender system algorithms on data with varying characteristics. In: Ng, W.K., Kitsuregawa, M., Li, J., Chang, K. (eds.) Advances in Knowledge Discovery and Data Mining. 10th Pacific-Asia Conf. (PAKDD’06), Singapore, April 9–12 (2006)
Van Setten, M.: Supporting people in finding information: hybrid recommender systems and goal-based structuring. PhD thesis, Telematica Inst., Netherlands, Fund. Res. Ser. 016 (2005)
Vincke, P.: Multicriteria Decision-Aid. J. Wiley, New York (1992)
Vuorikari, R., Manouselis, N., Duval, E.: Using metadata for storing, sharing, and reusing evaluations in social recommendation: the case of learning resources. In: Go, D.H., Foo, S. (eds.) Social Information Retrieval Systems: Emerging Technologies and Applications for Searching the Web Effectively. Idea Group Publishing, Hershey, PA, in press.
Wang, F.-H., Shao, H.-M.: Effective personalized recommendation based on time-framed navigation clustering and association mining. Exp. Syst. Appl. 27, 365–377 (2004)
Wang, P.: Recommendation based on personal preference. In: Zhang, Y., Kandel, A., Lin, T., Yao, Y. (eds.) Comput. Web Intell.: Intell. Techn. Web Appl., World Scientific (2004)
Wang, Y.-F., Chuang, Y.-L., Hsu, M.-H., Keh, H.-C.: A personalized recommender system for the cosmetic business. Exp. Syst. Appl. 26, 427–434 (2004)
Wei, C.-P., Shaw, M.J., Easley, R.F.: A survey of recommendation systems in electronic commerce. In: Rust, R.T., Kannan, P.K. (eds.) E-Serv.: New Dir. in Theor. and Pract., M. E. Sharpe (2002)
Yano, E., Sueyoshi, E., Shinohava, I., Kato, T.: Development of a recommendation system with multiple subjective evaluation process models. In Proc. 2nd Int. Conf. Cyberworlds, Singapore, December. IEEE Computer Society, Washington, DC (2003)
Yu, C.-C.: Designing a web-based consumer decision support systems for tourism services. In Proc. 4th Int. Conf. Electr. Comm., Hong Kong, October (2002)
Yu, C.-C.: Innovation, management & strategy: a web-based consumer-oriented intelligent decision support system for personalized e-services. In Proc. 6th Int. Conf. Electr. Comm., March. ACM, New York (2004)
Yuan, S.-T., Cheng, C.: Ontology-based personalized couple clustering for heterogeneous product recommendation in mobile marketing. Exp. Syst. Appl. 26, 461–476 (2004)
Yuan, S.-T., Tsao, Y.W.: A recommendation mechanism for contextualized mobile advertising. Exp. Syst. Appl. 24, 399–414 (2003)
Zeleny, M.: Linear Multiobjective Programming. Springer, Berlin Heidelberg New York (1974)
Zimmerman, J., Kurapati, K., Buczak, A.L., Schaffer, D., Martino, J., Gutta, S.: TV personalization system: design of a TV show recommender engine and interface. In: Ardissono, L., Kobsa, A., Maybury, M. (eds.) Personalized Digital Television: Targeting Programs to Individual Viewers, vol. 6. Hum.-Comp. Inter. Ser., Kluwer, Dordrecht (2004)
Zimmerman, J., Parameswaran, L., Kurapati, K.: Celebrity recommender. In Proc. AH’2002 Work. “Recommendation and Personalization in eCommerce”, Málaga, Spain, May 28th (2002)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Manouselis, N., Costopoulou, C. Analysis and Classification of Multi-Criteria Recommender Systems. World Wide Web 10, 415–441 (2007). https://doi.org/10.1007/s11280-007-0019-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11280-007-0019-8