Offline performance vs. subjective quality experience: a case study in video game recommendation

Research in the field of recommender systems is largely based on offline experimentation on historical datasets. Several recent works however suggest that models optimized for accuracy measures are not necessarily those that lead to the best user experience or perceived system utility. In this work we first determine the offline performance of different algorithms in the domain of video game recommendation and then investigate the perceived recommendation quality through a user study. The offline results show that learning-to-rank methods optimized for implicit feedback situations as expected perform best in terms of accuracy, where higher accuracy often comes with a stronger tendency of the algorithms to recommend mostly popular items. In the user study, however, methods that also consider the similarity between items in their algorithms perform at least equally well in terms of accuracy, which could not be expected from the offline experiment. Such content-enhanced methods were also slightly favored by users in terms of perceived transparency.