There is a method to it: Making meaning in information research through a mix of paradigms and methods.
•Download as PPTX, PDF•
0 likes•116 views
This document summarizes a presentation on using mixed methods approaches in information behavior research. It discusses how log analysis of search queries can be combined with user interviews to provide context around search behaviors. As an example, the presentation examines a project that analyzed WorldCat Discovery search logs and conducted interviews with users to better understand how they navigate from search to accessing information. The methodology provides benefits like context around quantitative log data, but also challenges in terms of resources needed. Overall it argues for the value of mixed methods approaches in gaining a richer understanding of user behaviors and experiences.
There is a method to it: Making meaning in information research through a mix of paradigms and methods.
- 1. Melbourne, Australia • ASIS&T • 21 October 2019 There is a Method to It: Making Meaning in Information Research through a Mix of Paradigms and Methods Lynn Silipigni Connaway, Ph.D. Director of Library Trends and User Research, OCLC Ixchel Faniel, Ph.D. Senior Research Scientist, OCLC Bhuva Narayan, Ph.D. Transdisciplinary Researcher, UTS Elham Sayyad-Abdi, Ph.D. Information Researcher
- 2. “Survey research is one of the most popular methods used in the LIS literature.” (Case & Given, 2016) Prevalence of Survey Research Image: https://www.flickr.com/photos/froderik/9355085596/ by Fredrik Rubensson/CC BY-SA 2.0
- 3. “Perhaps the most convenient method…will be to follow the reader from the moment he enters the library to the moment he leaves it…” (Ranganathan, 1931, 337) Image: https://www.flickr.com/photos/anjan58/7346141798 by anjan58 / CC BY-NC-ND 2.0
- 4. • Interviews, 51 • Surveys, 34 • Content Analysis, 28 (Greifeneder, 2014) Information Behaviour Research Methods: JASIST, Information Research 2012-2013 JDOC, iConference Proceedings 2013-2014 Image:https://www.flickr.com/photos/kmakice/4390520785/by Kevin Makice/CC BY-NC-SA 2.0
- 5. Information Behaviour Research Methods: JASIST, Information Research 2012-2013 JDOC, iConference Proceedings 2013-2014 • Methods used less than five times: • Delphi studies (Poirier & Robinson, 2014) • Eye-tracking (e.g. Balatsoukas & Ruthven, 2012; Wildemuth, 2009) • Log file analysis (Jiang, 2014) • Participatory designs (Greifeneder, 2014) Image:https://www.flickr.com/photos/novecentino/50011555/by Gorgio Montersino/CC BY-SA 2.0
- 6. • Mixed Methods, 45% • More than two methods, 7% • Qualitative-Qualitative, 69% • Quantitative-Qualitative, 31% • Quantitative-Quantitative, 0% (Greifeneder, 2014) Information Behaviour Research Methods: JASIST, Information Research 2012-2013 JDOC, iConference Proceedings 2013-2014 Image:https://www.flickr.com/photos/csb13/37236019891/by Chris Blakeley/CC BY-NC-ND 2.0
- 7. Qualitative and mixed-method studies are not as prevalent in information behavior studies but demonstrate a greater variety of research strategies and data collection techniques. (Matusiak, 2017; McKechnie et al., 2002) Image:https://www.flickr.com/photos/gawthrop/8274121366/by Peter Gawthrop/CC BY-NC 2.0
- 8. Call for Qualitative and Mixed Methods Research • Rich data and thick description • Answer how and why questions • Can support decision-making for resources and programs • Strategic planning from a user perspective • Can help motivate and increase buy-in for change • Provides opportunity to build relationships (Connaway & Radford, 2017; Asher & Miller, 2011) Image: https://www.flickr.com/photos/ihardlyflickr/2253071281/ by Mason Bryant/CC BY-SA 2.0
- 9. Research Methodology Discovery system logs combined with semi-structured interviews Image:https://www.flickr.com/photos/gawthrop/8274084522/by Peter Gawthrop/CC BY-NC 2.0
- 10. Theoretical Background • Log analysis to collect large amounts of unbiased user data (Jansen 2006, Connaway and Radford 2017) • Logs used to study how people use online systems • Catalog Search failure rates • Behavior of digital library users • Use of e-journals • User experience with video and music streaming services (Hunter 1991; Jamali, Nicholas, and Huntington 2005; Lamkhede and Das 2019; Nouvellet, et al. 2019) Image: https://www.flickr.com/photos/gawthrop/8272965471/by Peter Gawthrop/CC BY-NC 2.0
- 11. Theoretical Background • Problems with log analysis • Ambiguity of log events • Actions not captured in logs • Combining log analysis with user interviews • Asked questions about search and analyzed transaction logs (Connaway, Budd, and Kochtanek 1995) • No indication that combining search logs with individual interviews has been used since Image:https://www.flickr.com/photos/gawthrop/8273046981/ by Peter Gawthrop/CC BY-NC 2.0
- 12. Discovery and Access Project: How do academic library users navigate the path from discovery to access? • What do academic users do when searches don't result in fulfillment? • What differentiates searches that lead to access from searches that don’t? • What demographic characteristics influence the access of users? • How does access correlate with success? Image:https://www.flickr.com/photos/gawthrop/8274042600/by Peter Gawthrop/CC BY-NC 2.0
- 13. WORLDCAT DISCOVERY SEARCH LOG ANALYSIS “Log analysis is everything that a lab study is not.” (Jansen 2017, 349)
- 14. 1. Did a keyword search but mistyped it - Had 0 results 2. Redid keyword search with correct spelling - Had 759,902 results 3. Began typing in additional keyword 4. Selected one of the autosuggested keyword phrases - Had 1,761 results What do the raw logs tell us?
- 15. Ways of evolving a search Corrected search Refined search Shows greater than 90% similarity with the previous search string Shows 80–90% similarity with the previous search string, with the first string contained in the second, or an index change Shows less than 80% similarity with the previous search string New search
- 16. Summary of results • Average of 5 minutes per session • Average of 2.2 searches per session • Average of 5.1 words per search • 12% of sessions had search refinements • 33% of sessions had multiple searches n=282,307 sessions Image:https://www.flickr.com/photos/claudine/8344927130/claundice/CC BY-SA 2.0
- 17. Types of Requests Search results Physical access options Online access attempt Attempt to save Physical access attempt The user made a request for search results. This could include a new search, refinement of an existing search, or the addition of limiters. The user clicked an item or made a request to digitally access the full text of the item. The user attempted to export or otherwise save the citation. The user clicked an item or made a request to place a hold on a physical copy of the item. Some users left the system after looking at holdings, where they were able to identify the physical item call number and/or location. These users were categorized as having the option to physically access the item.
- 18. USER INTERVIEWS “User interviews can help capture search and discovery behavior as the user understands it, rather than as a computer system understands it.” (Connaway, Cyr, Brannon, Gallagher, and Hood 2019)
- 19. Example questions • “Please tell us what you were looking for and why you decided to do an online search.” • “Did the item you were searching for come up in your search results? In other words, did you find it?” • “I’d like to understand how you felt about your search experience overall. Would you say you were delighted with your search experience?” Image:https://www.flickr.com/photos/mattknox/2842537496/ by Matthewknox/CC BY-NC-ND 2.0
- 20. What the logs told us: • Began keyword search but mistyped it o Had 0 results • Redid keyword search with correct spelling o Had 759,902 results • Began typing in additional keyword • Selected one of the autosuggested phrases o Had 1,761 results What do the interviews tell us?
- 21. What the logs told us: • Began keyword search but mistyped it o Had 0 results • Redid keyword search with correct spelling o Had 759,902 results • Began typing in additional keyword • Selected one of the autosuggested phrases o Had 1,761 results • Just starting work on a paper on a broad topic; didn’t yet have a direction for the paper • Was overwhelmed with number of search results • Abandoned “library search” to do “Google searching” to better determine a direction for the paper • Later came back to the library search and found it useful • Also received help from student workers in the library • Felt “prepared” to use the library search due to 1st-year library instruction What do the interviews tell us?
- 22. Interview coding themes Item formats Search strategies Decision-making factors Liked or desired features Evaluation of resources Feelings of frustration and delight Influence of librarian
- 23. METHODOLOGY CHALLENGES AND BENEFITS “The methodology used for this study also could be extended beyond discovery systems. Other computerized activities that leave digital traces could be studied using interview protocols based on log analysis.” (Connaway, Cyr, Brannon, Gallagher, and Hood 2019)
- 24. Challenges of methodology (Tandem use of log data and user interviews) • Resource intensive Time consuming Multiple team members Multiple IRBs • High level of expertise required Image:https://www.flickr.com/photos/queenscollege/3309770091/ by Queen’s College/CC BY 2.0
- 25. Benefits of methodology (Tandem use of log data and user interviews) • Provide context for quantitative data • Clarify qualitative data • Most effective when digital traces are present • Inform development of literacy instruction Image:https://www.flickr.com/photos/pipwilson/2347270176/by Pip Wilson/CC BY-NC-ND 2.0
- 26. ACKNOWLEDGEMENTS We would like to thank the librarians and library users who participated in this research. We also thank additional OCLC staff who assisted with the project: Jay Holloway and Ralph LeVan for assisting with data collection and analysis and Nick Spence for his assistance in preparing this presentation.
- 27. References Asher, A., & Miller, S. (2011). So you want to do anthropology in your library? Or a practical guide to ethnographic research in academic libraries. Chicago: The ERIAL Project. Case, D. O., & Given, L. S. (2016). Looking for information: A survey of research on information seeking, needs, and behaviour. Bingley, UK: Emerald. Connaway, Lynn Silipigni, John M. Budd, and Thomas R. Kochtanek. 1995. “An Investigation of the Use of an Online Catalog: User Characteristics and Transaction Log Analysis.” Library Resources and Technical Services 39, no. 2: 142–152. Connaway, Lynn Silipigni, Chris Cyr, Brittany Brannon, Peggy Gallagher, and Erin Hood. 2019. “Speaking on the Record: Combining Interviews with Search Log Analysis in User Research.” ALISE/ProQuest Methodology Paper Competition Award Winner. Connaway, Lynn Silipigni, and Marie L. Radford. 2017. Research Methods in Library and Information Science, 6th ed. Santa Barbara, CA: Libraries Unlimited. Connaway, Lynn Silipigni, and Marie L. Radford. 2018. Survey Research. Webinar presented by ASIS&T, January 23. https://www.slideshare.net/LynnConnaway/survey-research-methods-with-lynn-silipigni-connaway and https://www.youtube.com/watch?v=4dlpAT7MXh0 . Greifeneder, E. (2014). Trends in information behaviour research. In Proceedings of ISIC, the Information Behaviour Conference, Leeds, 2-5 September, 2014: Part 1. http://InformationR.net/ir/19-4/isic/isic13.html Hunter, Rhonda N. 1991. “Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library: A Transaction Log Analysis.” RQ 30, no. 3: 395–402. https://www.jstor.org/stable/25828813.
- 28. References Jamali, Hamid R., David Nicholas, and Paul Huntington. 2005. “The Use and Users of Scholarly E-Journals: A Review of Log Analysis Studies.” Aslib Proceedings 57, no. 5: 554–571. https://doi.org/10.1108/00012530510634271. Jansen, Bernard J. 2006. “Search Log Analysis: What It Is, What’s Been Done, How to Do It.” Library and Information Science Research 28, no. 3: 407–432. Jansen, Bernard J. 2017. “Log Analysis.” In Research Methods for Library and Information Science, 6th ed., edited by Lynn Silipigni Connaway and Marie L. Radford, 348-349. Santa Barbara, CA: Libraries Unlimited. Lamkhede, Sudarshan, and Sudeep Das. 2019. “Challenges in Search on Streaming Services: Netflix Case Study.” In the Proceedings of SIGIR ’19, July 21–25, 2019, Paris, France. https://arxiv.org/pdf/1903.04638.pdf. Matusiak, K. K. (2017). Studying information behavior of image users: An overview of research methodology in LIS literature, 2004-2015. Library and Information Science Research 39 (1), 53–60. McKechnie, L. M., Baker, L., Greenwood, M., & Julien, H. (2002). Research method trends in human information behaviour literature. The New Review of Information Behaviour Research: Studies of Information Seeking in Context (Proceedings of ISIC 2002), 3, 113-125. Nouvellet, Adrien, Florence D’Alché-Buc, Valérie Baudouin, Christophe Prieur, and François Roueff. 2019. “A Quantitative Analysis of Digital Library User Behaviour Based on Access Logs.” Qualitative and Quantitative Methods in Libraries 7, no. 1: 1–13. Ranganathan, S. R. (1931). The five laws of library science. London: Edward Goldston, Ltd.
- 29. Questions & Discussion Lynn Silipigni Connaway, PhD connawal@oclc.org @LynnConnaway Ixchel Faniel, Ph.D. Senior Research Scientist, OCLC Elham Sayyad-Abdi, Ph.D. Information Researcher Bhuva Narayan, Ph.D. Transdisciplinary Researcher, UTS