Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13185))

Included in the following conference series:

Abstract

This work explores the reproducibility of CFGAN. CFGAN and its family of models (TagRec, MTPR, and CRGAN) learn to generate personalized and fake-but-realistic rankings of preferences for top-N recommendations by using previous interactions. This work successfully replicates the results published in the original paper and discusses the impact of certain differences between the CFGAN framework and the model used in the original evaluation. The absence of random noise and the use of real user profiles as condition vectors leaves the generator prone to learn a degenerate solution in which the output vector is identical to the input vector, therefore, behaving essentially as a simple autoencoder. The work further expands the experimental analysis comparing CFGAN against a selection of simple and well-known properly optimized baselines, observing that CFGAN is not consistently competitive against them despite its high computational cost. To ensure the reproducibility of these analyses, this work describes the experimental methodology and publishes all datasets and source code.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The supplemental material [29] contains the formal formulation of GANs.

  2. 2.

    For a detailed explanation of CFGAN we refer the reader to the reference article [4].

  3. 3.

    https://github.com/recsyspolimi/ecir-2022-an-evaluation-of-GAN-for-CF and [29].

  4. 4.

    The Watcha [4] dataset was not provided with the reference article.

  5. 5.

    The reference article does not provide instructions to reproduce this version of the dataset. We contacted the authors for clarifications but did not receive a reply.

  6. 6.

    A table with all metrics is available in the supplemental materials of this work.

  7. 7.

    The full results are in the supplemental material [29].

References

  1. Adomavicius, G., Kwon, Y.: Improving aggregate recommendation diversity using ranking-based techniques. IEEE Trans. Knowl. Data Eng. 24(5), 896–911 (2012). https://doi.org/10.1109/TKDE.2011.15

    Article  Google Scholar 

  2. Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, 2–6 November 2009, pp. 601–610. ACM (2009). https://doi.org/10.1145/1645953.1646031

  3. Borji, A.: Pros and cons of GAN evaluation measures. Comput. Vis. Image Underst. 179, 41–65 (2019). https://doi.org/10.1016/j.cviu.2018.10.009

    Article  Google Scholar 

  4. Chae, D., Kang, J., Kim, S., Lee, J.: CFGAN: a generic collaborative filtering framework based on generative adversarial networks. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, 22–26 October 2018, pp. 137–146. ACM (2018). https://doi.org/10.1145/3269206.3271743

  5. Chen, H., Wang, S., Jiang, N., Li, Z., Yan, N., Shi, L.: Trust-aware generative adversarial network with recurrent neural network for recommender systems. Int. J. Intell. Syst. 36(2), 778–795 (2021). https://doi.org/10.1002/int.22320

    Article  Google Scholar 

  6. Christoffel, F., Paudel, B., Newell, C., Bernstein, A.: Blockbusters and wallflowers: accurate, diverse, and scalable recommendations with random walks. In: Proceedings of the 9th ACM Conference on Recommender Systems, RecSys 2015, Vienna, Austria, 16–20 September 2015, pp. 163–170. ACM (2015). https://doi.org/10.1145/2792838.2800180

  7. Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-N recommendation tasks. In: Proceedings of the 2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, 26–30 September 2010, pp. 39–46. ACM (2010). https://doi.org/10.1145/1864708.1864721

  8. Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A.A.: Generative adversarial networks: an overview. IEEE Signal Process. Mag. 35(1), 53–65 (2018). https://doi.org/10.1109/MSP.2017.2765202

    Article  Google Scholar 

  9. Fellicious, C., Weissgerber, T., Granitzer, M.: Effects of random seeds on the accuracy of convolutional neural networks. In: Nicosia, G., et al. (eds.) LOD 2020, Part II. LNCS, vol. 12566, pp. 93–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64580-9_8

    Chapter  Google Scholar 

  10. Ferrari Dacrema, M., Boglio, S., Cremonesi, P., Jannach, D.: A troubling analysis of reproducibility and progress in recommender systems research. ACM Trans. Inf. Syst. 39(2), 20:1–20:49 (2021). https://doi.org/10.1145/3434185

  11. Ferrari Dacrema, M., Cremonesi, P., Jannach, D.: Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In: Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, 16–20 September 2019, pp. 101–109. ACM (2019). https://doi.org/10.1145/3298689.3347058

  12. Ferrari Dacrema, M., Parroni, F., Cremonesi, P., Jannach, D.: Critically examining the claimed value of convolutions over user-item embedding maps for recommender systems. In: CIKM 2020: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, 19–23 October 2020, pp. 355–363. ACM (2020). https://doi.org/10.1145/3340531.3411901

  13. Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, Quebec, Canada, 8–13 December 2014, pp. 2672–2680 (2014). https://proceedings.neurips.cc/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html

  14. Goodfellow, I.J., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020). https://doi.org/10.1145/3422622

    Article  MathSciNet  Google Scholar 

  15. Harper, F.M., Konstan, J.A.: The MovieLens datasets: history and context. ACM Trans. Interact. Intell. Syst. 5(4), 19:1–19:19 (2016). https://doi.org/10.1145/2827872

  16. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 5967–5976. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.632

  17. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 4401–4410. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00453

  18. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 8107–8116. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00813

  19. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)

    Google Scholar 

  20. Kingma, D.P., Welling, M.: An introduction to variational autoencoders. Found. Trends Mach. Learn. 12(4), 307–392 (2019). https://doi.org/10.1561/2200000056

    Article  MATH  Google Scholar 

  21. Lin, J.: The neural hype and comparisons against weak baselines. SIGIR Forum 52(2), 40–51 (2019). https://doi.org/10.1145/3308774.3308781

  22. Lin, J.: The neural hype, justified! A recantation. SIGIR Forum 53(2), 88–93 (2021). https://doi.org/10.1145/3458553.3458563

  23. Lipton, Z.C., Steinhardt, J.: Troubling trends in machine learning scholarship. ACM Queue 17(1), 80 (2019). https://doi.org/10.1145/3317287.3328534

    Article  Google Scholar 

  24. Ludewig, M., Jannach, D.: Evaluation of session-based recommendation algorithms. User Model. User Adapt. Interact. 28(4-5), 331–390 (2018). https://doi.org/10.1007/s11257-018-9209-6

  25. Madhyastha, P., Jain, R.: On model stability as a function of random seed. In: Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, 3–4 November 2019, pp. 929–939. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/K19-1087

  26. Mirza, M., Osindero, S.: Conditional generative adversarial nets. CoRR abs/1411.1784 (2014). http://arxiv.org/abs/1411.1784

  27. Moreno-Torres, J.G., Raeder, T., Alaíz-Rodríguez, R., Chawla, N.V., Herrera, F.: A unifying view on dataset shift in classification. Pattern Recognit. 45(1), 521–530 (2012). https://doi.org/10.1016/j.patcog.2011.06.019

    Article  Google Scholar 

  28. Ning, X., Karypis, G.: SLIM: sparse linear methods for top-N recommender systems. In: 11th IEEE International Conference on Data Mining, ICDM 2011, Vancouver, BC, Canada, 11–14 December 2011, pp. 497–506. IEEE Computer Society (2011). https://doi.org/10.1109/ICDM.2011.134

  29. Pérez Maurera, F.B., Ferrari Dacrema, M., Cremonesi, P.: An Evaluation of Generative Adversarial Networks for Collaborative Filtering - Supplemental Material (2022). https://doi.org/10.5281/zenodo.5879345

  30. Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D.: Dataset Shift in Machine Learning. The MIT Press, New York (2009)

    Google Scholar 

  31. Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian personalized ranking from implicit feedback. In: UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, 18–21 June 2009, pp. 452–461. AUAI Press (2009)

    Google Scholar 

  32. Steck, H.: Embarrassingly shallow autoencoders for sparse data. In: The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, 13–17 May 2019, pp. 3251–3257. ACM (2019). https://doi.org/10.1145/3308558.3313710

  33. Tang, J., Gao, H., Liu, H.: mTrust: discerning multi-faceted trust in a connected world. In: Proceedings of the Fifth International Conference on Web Search and Web Data Mining, WSDM 2012, Seattle, WA, USA, 8–12 February 2012, pp. 93–102. ACM (2012). https://doi.org/10.1145/2124295.2124309

  34. Wang, J., et al.: IRGAN: a minimax game for unifying generative and discriminative information retrieval models. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, 7–11 August 2017, pp. 515–524. ACM (2017). https://doi.org/10.1145/3077136.3080786

  35. Wang, Z., Xu, Q., Ma, K., Jiang, Y., Cao, X., Huang, Q.: Adversarial preference learning with pairwise comparisons. In: Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, 21–25 October 2019, pp. 656–664. ACM (2019). https://doi.org/10.1145/3343031.3350919

  36. Xia, B., Bai, Y., Yin, J., Li, Q., Xu, L.: MTPR: a multi-task learning based poi recommendation considering temporal check-ins and geographical locations. Appl. Sci. 10(19) (2020). https://doi.org/10.3390/app10196664

  37. Xie, F., Li, S., Chen, L., Xu, Y., Zheng, Z.: Generative adversarial network based service recommendation in heterogeneous information networks. In: 2019 IEEE International Conference on Web Services, ICWS 2019, Milan, Italy, 8–13 July 2019, pp. 265–272. IEEE (2019). https://doi.org/10.1109/ICWS.2019.00053

  38. Yang, W., Lu, K., Yang, P., Lin, J.: Critically examining the “neural hype”: weak baselines and the additivity of effectiveness gains from neural ranking models. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, 21–25 July 2019, pp. 1129–1132. ACM (2019). https://doi.org/10.1145/3331184.3331340

  39. Zhou, T., Kuscsik, Z., Liu, J.G., Medo, M., Wakeling, J.R., Zhang, Y.C.: Solving the apparent diversity-accuracy dilemma of recommender systems. Proc. Natl. Acad. Sci. 107(10), 4511–4515 (2010). https://doi.org/10.1073/pnas.1000488107

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Politecnico di Milano, Milan, Italy

    Fernando Benjamín Pérez Maurera, Maurizio Ferrari Dacrema & Paolo Cremonesi

  2. ContentWise, Milan, Italy

    Fernando Benjamín Pérez Maurera

Authors
  1. Fernando Benjamín Pérez Maurera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maurizio Ferrari Dacrema
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paolo Cremonesi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fernando Benjamín Pérez Maurera .

Editor information

Editors and Affiliations

  1. Martin Luther University Halle-Wittenberg, Halle, Germany

    Matthias Hagen

  2. Leiden University, Leiden, The Netherlands

    Suzan Verberne

  3. University of Glasgow, Glasgow, UK

    Craig Macdonald

  4. University of Duisburg-Essen, Essen, Germany

    Christin Seifert

  5. University of Stavanger, Stavanger, Norway

    Krisztian Balog

  6. Norwegian University of Science and Technology, Trondheim, Norway

    Kjetil Nørvåg

  7. University of Stavanger, Stavanger, Norway

    Vinay Setty

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pérez Maurera, F.B., Ferrari Dacrema, M., Cremonesi, P. (2022). An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering. In: Hagen, M., et al. Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13185. Springer, Cham. https://doi.org/10.1007/978-3-030-99736-6_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99736-6_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99735-9

  • Online ISBN: 978-3-030-99736-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation