article

Task complexity and difficulty in music information retrieval

Authors:

Noriko KandoAuthors Info & Claims

Journal of the Association for Information Science and Technology, Volume 68, Issue 7

Pages 1711 - 1723

https://doi.org/10.1002/asi.23803

Published: 01 July 2017 Publication History

Abstract

There has been little research on task complexity and difficulty in music information retrieval MIR, whereas many studies in the text retrieval domain have found that task complexity and difficulty have significant effects on user effectiveness. This study aimed to bridge the gap by exploring i the relationship between task complexity and difficulty; ii factors affecting task difficulty; and iii the relationship between task difficulty, task complexity, and user search behaviors in MIR. An empirical user experiment was conducted with 51 participants and a novel MIR system. The participants searched for 6 topics across 3 complexity levels. The results revealed that i perceived task difficulty in music search is influenced by task complexity, user background, system affordances, and task uncertainty and enjoyability; and ii perceived task difficulty in MIR is significantly correlated with effectiveness metrics such as the number of songs found, number of clicks, and task completion time. The findings have implications for the design of music search tasks in research or use cases in system development as well as future MIR systems that can detect task difficulty based on user effectiveness metrics.

References

[1]

Arguello, J., Wu, W.C., Kelly, D., &Edwards, A. 2012. Task complexity, vertical display and user interaction in aggregated search. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR p. pp.435. Portland, Oregon, Aug. 2012. New York, NY: ACM.

Digital Library

[2]

Aula, A., Khan, R.M., &Guan, Z. 2010. How does search behavior change as search becomes more difficult? In Proceedings of the 28th International SIGCHI Conference on Human Factors in Computing Systems pp. pp.35-44. Atlanta, Georgia, Apr. 2010. New York, NY: ACM.

Digital Library

[3]

Bailey, P., Moffat, A., Scholer, F., &Thomas, P. 2015 User variability and IR system evaluation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR 2015 pp. 625-634. Santiago, Chile, Aug. 2015. New York, NY: ACM.

Digital Library

[4]

Byström, K., &Järvelin, K. 1995. Task complexity affects information seeking and use. Information Processing & Management, Volume 31 Issue 2, pp.191-213.

Digital Library

[5]

Casey, M., Veltkamp, R., Goto, M., Leman, M., Rhodes, C., &Slaney, M. 2008. Content-based music information retrieval: Current directions and future challenges. Proceedings of the IEEE, Volume 96 Issue 4, pp.668-696.

[6]

Corder, G.W., &Foreman, D.I. 2009. Nonparametric statistics for non-statisticians: A step-by-step approach. Hoboken, NJ: John Wiley & Sons.

[7]

Downie, J.S. 2003. Music information retrieval. Annual Review of Information Science and Technology, Volume 37 Issue 1, pp.295-340.

[8]

Downie, J.S., &Cunningham, S.J. 2002. Toward a theory of music information retrieval queries: System design implications. In Proceedings of the 3rd International Conference on Music Information Retrieval ISMIR pp. 299-300. Paris, France, Oct. 2002.

[9]

Downie, J.S., Hu, X., Lee, J.H., Choi, K., Cunningham, S.J., Hao, Y., &Bainbridge, D. 2014 Ten years of MIREX Music Information Retrieval Evaluation eXchange: Reflections, challenges and opportunities. In Proceedings of the 15th International Conference on Music Information Retrieval ISMIR. Taipei, Taiwan Oct. 2014.

[10]

Haerem, T., &Rau, D. 2007. The influence of degree of expertise and objective task complexity on perceived task complexity and performance. Journal of Applied Psychology, Volume 925, pp.1320.

[11]

Hu, X., Downie, J.S., Laurier, C., Bay, M., &Ehmann, A. 2008. The 2007 MIREX audio mood classification task: Lessons learned. In Proceedings of the 9th International Conference on Music Information Retrieval ISMIR pp. pp.462-467. Philadelphia, PA, Sept. 2008.

[12]

Hu, X., &Kando, N. 2012. User-centered measures vs. system effectiveness in finding similar songs. In Proceedings of the 13th International Conference on Music Information Retrieval ISMIR pp. pp.331-336. Porto, Portugal, Oct. 2012.

[13]

Hu, X., &Kando, N. 2014 Evaluation of music search in casual-leisure situations. In Proceedings of Search for Fun Workshop at the Information Interaction in Context conference IIiX, Aug. 2014, Regensburg, Germany.

[14]

Hu, X., Kando, N., &Yuan, X. 2011. User evaluation of an interactive music information retrieval system. In Proceedings of the 5th Workshop on Human-Computer Interaction and Information Retrieval HCIR, Mountain View, CA

[15]

Hu, X., Lee, J.H., Bainbridge, D., Choi, K., Organisciak, P., &Downie, J.S. 2015. The MIREX grand challenge: A framework of holistic user-experience evaluation in music information retrieval. Journal of the Association for Information Science and Technology.

Digital Library

[16]

Hu, X., &Liu, J. 2010. Evaluation of music information retrieval: Towards a user-centered approach. In Proceedings of the 4th Workshop on Human-Computer Interaction and Information Retrieval HCIR, New Brunswick, NJ, Aug. 2010

[17]

Hu, X., Sanghvi, V., Vong, B., On, P.J., Leong, C., &Angelica, J. 2008. Moody: A web-based music mood classification and recommendation system. In Proceedings of the 9th International Conference on Music Information Retrieval ISMIR, Philadelphia, PA, Sept. 2008.

[18]

Iwamura, R. 2001. U.S. Patent No. 6,188,010. Washington, DC: U.S. Patent and Trademark Office.

[19]

Kelly, D., Arguello, J., Edwards, A., &Wu, W.C. 2015. Development and evaluation of search tasks for IIR experiments using a cognitive complexity framework. In Proceedings of the 2015 International Conference on the Theory of Information Retrieval ICTIR 2015 pp. pp.101-110.

Digital Library

[20]

Kim, J. 2006. Task difficulty as a predictor and indicator of web searching interaction. In CHI'06 Extended Abstracts on Human Factors in Computing Systems pp. pp.959-964. Montreal, Canada, Apr. 2006. New York, NY: ACM.

Digital Library

[21]

Krathwohl, D.R. 2002. A revision of Bloom's taxonomy: An overview. Theory into Practice, Volume 41 Issue 4, pp.212-218.

[22]

Laplante, A., &Downie, J.S. 2011. The utilitarian and hedonic outcomes of music information-seeking in everyday life. Library and Information Science Research, Volume 33 Issue 3, pp.202-210.

[23]

Lee, J.H. 2010. Analysis of user needs and information features in natural language queries seeking music information. Journal of the American Society for Information Science and Technology, Volume 61 Issue 5, pp.1025-1045.

Digital Library

[24]

Lee, J.H., &Cunningham, S.J. 2013. Toward an understanding of the history and impact of user studies in music information retrieval. Journal of Intelligent Information Systems, Volume 41 Issue 3, pp.499-521.

Digital Library

[25]

Lee, J.H., &Downie, J.S. 2004. Survey of music information needs, uses, and seeking behaviours: Preliminary findings. In Proceedings of the 5th International Conference on Music Information Retrieval ISMIR pp. pp.441-446. Barcelona, Spain, Oct. 2004.

[26]

Lee, J.H., &Waterman, N.M. 2012. Understanding user requirements for music information services. In Proceedings of the 13th International Conference on Music Information Retrieval ISMIR pp. pp.253-258. Porto, Portugal, Oct. 2012.

[27]

Levitin, D.J. 2011. This is your brain on music: Understanding a human obsession. London, UK: Atlantic Books.

[28]

Liu, J. 2015. User assessment of search task difficulty: Relationships between reasons and ratings. Library & Information Science Research Volume 37, pp.329-337.

[29]

Liu, J., Liu, C., Cole, M., Belkin, N.J., &Zhang, X. 2012. Exploring and predicting search task difficulty. In Proceedings of the 21st ACM international conference on Information and knowledge management pp. pp.1313-1322. Maui, HI, Oct. 2012. New York, NY: ACM.

Digital Library

[30]

Liu, J., Liu, C., Gwizdka, J., &Belkin, N.J. 2010. Can search systems detect users' task difficulty? Some behavioral signals. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval pp. pp.845-846. Geneva, Switzerland, July 2010. New York, NY: ACM.

Digital Library

[31]

Norman, D.A. 1988. The psychology of everyday things. New York: Basic Books paperback as the Design of Everyday Things. New York: Doubleday, pp.1990.

[32]

Salaba, A., &Zhang, Y. 2012. Searching for music: End-user perspectives on system features. In D.R.Neal Ed., Indexing and retrieval of non-text information pp. pp.137-159. Berlin: De Gruyter.

[33]

Schedl, M., Gómez, E., &Urbano, J. 2014. Music information retrieval: Recent developments and applications. Foundations and Trends in Information Retrieval, Volume 82-3, pp.127-261.

Digital Library

[34]

Vignoli, F. 2004. Digital music interaction concepts: A user study. ISMIR. In Proceedings of the 5th International Conference on Music Information Retrieval ISMIR. Barcelona, Spain, Oct. 2004.

[35]

Weninger, F., Wöllmer, M., &Schuller, B. 2011. Automatic assessment of singer traits in popular music: Gender, age, height and race. In Proceedings of the International Society for Music Information Retrieval Conference ISMIR. Miami, FL, USA.

[36]

White, R.W., &Dumais, S.T. 2009. Characterizing and predicting search engine switching behavior. In Proceedings of the 18th ACM conference on Information and Knowledge Management pp. pp.87-96. Hong Kong, China, Nov. 2009. New York, NY: ACM.

Digital Library

[37]

Wildemuth, B.M., &Freund, L. 2009. Search tasks and their role in studies of search behaviours. In Proceedings of 2009 Workshop on Human Computer Interaction and Information Retrieval HCIR 2009, Oct 23, 2009. Retrieved from "http://ils.unc.edu/searchtasks/publication/publication_1.pdf"

[38]

Wildemuth, B.M., Freund, L., &Toms, E.G. 2014. Untangling search task complexity and difficulty in the context of interactive information retrieval studies. Journal of Documentation, Volume 706, pp.1118-1140.

[39]

Witten, I., &Frank, E. 1999. Data mining: Practical machine learning tools and techniques with Java implementations. San Fransisco: Morgan Kaufmann.

Digital Library

Cited By

Urgo KArguello J(2022)Understanding the “Pathway” Towards a Searcher’s Learning ObjectiveACM Transactions on Information Systems10.1145/349522240:4(1-43)Online publication date: 11-Jan-2022
https://dl.acm.org/doi/10.1145/3495222
Zhang H(2021)Voice Keyword Retrieval Method Using Attention Mechanism and Multimodal Information FusionScientific Programming10.1155/2021/66628412021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/6662841
Deng SZhao AFu SLiu YFan WJiang Y(2020)Music-search behaviour on a social Q&A siteJournal of Information Science10.1177/016555151986160546:4(560-574)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1177/0165551519861605
Show More Cited By

Task complexity and difficulty in music information retrieval
1. Information systems

Recommendations

Exploring search task difficulty reasons in different task types and user knowledge groups

Development of a search task difficulty reason scheme.Relationship between task difficulty and task type: common and different reasons across task types.Relationship between task difficulty and user knowledge: common and different reasons among ...
An Investigation on Task Difficulty: Does Task Difficulty Depend on the Technology Used in Task Completion?
SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2

Previous research indicates that task difficulty (i.e., students' judgments on a task's complexity) impacts their task performance. However, whether students' perceived task difficulty changes depending on the technology they use when completing tasks is ...
Predicting task difficulty for different task types
ASIS&T '10: Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem - Volume 47

This paper reports our investigation of differences in users' behavior between difficult and easy search tasks, as well as how these differences vary with different types of tasks. We also report how behavioral predictors of task difficulty vary across ...

Comments

Information & Contributors

Information

Published In

cover image Journal of the Association for Information Science and Technology

Journal of the Association for Information Science and Technology Volume 68, Issue 7

July 2017

206 pages

ISSN:2330-1635

EISSN:2330-1643

Issue’s Table of Contents

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 01 July 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Urgo KArguello J(2022)Understanding the “Pathway” Towards a Searcher’s Learning ObjectiveACM Transactions on Information Systems10.1145/349522240:4(1-43)Online publication date: 11-Jan-2022
https://dl.acm.org/doi/10.1145/3495222
Zhang H(2021)Voice Keyword Retrieval Method Using Attention Mechanism and Multimodal Information FusionScientific Programming10.1155/2021/66628412021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/6662841
Deng SZhao AFu SLiu YFan WJiang Y(2020)Music-search behaviour on a social Q&A siteJournal of Information Science10.1177/016555151986160546:4(560-574)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1177/0165551519861605
Huang KChen JLiu CZhang LHuang RWu DMarchionini GHe DCunningham SHansen P(2020)A Comparative Study of the Relationship between the Subjective Difficulty, Objective Difficulty of Search Tasks and Search BehaviorsProceedings of the ACM/IEEE Joint Conference on Digital Libraries in 202010.1145/3383583.3398614(421-424)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1145/3383583.3398614
Que YZheng YHsiao JHu XHuang RWu DMarchionini GHe DCunningham SHansen P(2020)Exploring the Effect of Personalized Background Music on Reading ComprehensionProceedings of the ACM/IEEE Joint Conference on Digital Libraries in 202010.1145/3383583.3398543(57-66)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1145/3383583.3398543
Choi BWard ALi YArguello JCapra R(2019)The Effects of Task Complexity on the Use of Different Types of Information in a Search Assistance ToolACM Transactions on Information Systems10.1145/337170738:1(1-28)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1145/3371707
Choi BCapra RArguello JAzzopardi LHalvey MRuthven IJoho HMurdock VQvarfordt P(2019)The Effects of Working Memory during Search Tasks of Varying ComplexityProceedings of the 2019 Conference on Human Information Interaction and Retrieval10.1145/3295750.3298948(261-265)Online publication date: 8-Mar-2019
https://dl.acm.org/doi/10.1145/3295750.3298948
Capra RArguello JO'Brien HLi YChoi BCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)The Effects of Manipulating Task Determinability on Search Behaviors and OutcomesThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210047(445-454)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210047
Keyaki AMiyazaki J(2018)Analysis of Relevant Text Fragments for Different Search Task TypesInformation Retrieval Technology10.1007/978-3-030-03520-4_6(60-66)Online publication date: 28-Nov-2018
https://dl.acm.org/doi/10.1007/978-3-030-03520-4_6

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents