Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

User-centered assessment: Leveraging what you know and filling in the gaps.

•
•
Lynn Connaway
Lynn Connaway

Reuter, K., & Connaway, L. S. (2018). User-centered assessment: Leveraging what you know and filling in the gaps. Part 1 in 3-part webinar series, Evaluating and sharing your library's impact, presented by OCLC Research WebJunction, April 24, 2018.

1 of 43
Download to read offline
#libdata4impact Webinar Series: Evaluating and Sharing Your Library’s Impact Part 1: April 24 Kara Reuter User-centered Assessment: Leveraging What You Know and Filling in the Gaps Part 2: August 14 Linda Hofschire Digging into Assessment Data: Tips, Tricks, and Tools of the Trade Part 3: October 3 Melissa Bowles-Terry Take Action: Using and Presenting Research Findings to Make Your Case For more information: https://www.webjunction.org/news/ webjunction/webinar-series-research-assessment.html
Series Learner Guide Use alone or with others to apply what you’re learning between sessions. 13 pages of questions, activities, and resources. Customizable to meet your team’s needs!
• Research devoted exclusively to the challenges facing libraries and archives • Research Library Partnership includes working groups to collaborate with institutions on research and issues • Lifelong learning from WebJunction, for all library staff and volunteers • All connected through a global network of 16,000+ member libraries • Global and Regional Councils bring worldwide viewpoints together, informing and guiding the cooperative from their unique perspective.
#libdata4impact Research Library Partnership: Library Assessment Interest Group • The OCLC Research Library Partnership invites librarians at partner institutions to participate in a new Library Assessment Interest Group, beginning in April 2018. • This interest group will accompany and extend the Webinar Series: Evaluating and Sharing Your Library's Impact
Lynn Silipigni Connaway Senior Research Scientist and Director of User Research, OCLC Research connawal@oclc.org @LynnConnaway
#libdata4impact OCLC Research
#libdata4impact Some Initial Questions 1. What is your definition of assessment? 2. What comes to mind when you hear the term “assessment”? 3. What benefits do you see for assessment? 4. What are your concerns?
#libdata4impact Why Assessment? • Answers questions: • What do users/stakeholders want & need? • How can services/programs better meet needs? • Is what we do working? • Could we do better? • What are problem areas? • Traditional stats don’t tell whole story
#libdata4impact Importance of Assessment “Librarians are increasingly called upon to document and articulate the value of academic and research libraries and their contribution to institutional mission and goals.” (ACRL Value of Academic Libraries, 2010, p. 6) Association of College and Research Libraries. (2010) Value of Academic Libraries: A Comprehensive Research Review and Report. Researched by Megan Oakleaf. Chicago: Association of College and Research Libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf
#libdata4impact Steps in Assessment Process 1. Why? • Identify purpose 2. Who? • Identify team 3. How? • Choose model/approach/method 4. Commit! • Training/planning
#libdata4impact Outcomes Assessment Basics Outcomes: “The ways in which library users are changed as a result of their contact with the library’s resources and programs.” (ALA/ACRL. 1998) “Libraries cannot demonstrate institutional value to maximum effect until they define outcomes of institutional relevance and then measure the degree to which they attain them.” (Kaufman & Watstein, 2008, p. 227) ALA/ACRL. (1998). Task Force on Academic Library Outcomes Assessment Report. Kaufman, P., & Watstein, S. B. (2008). Library Value (Return on Investment, ROI) and the Challenge of Placing a Value on Public Services. Reference Services Review, 36(3), 226-231.
#libdata4impact Outputs & Inputs • Outputs • Quantify the work done • Don’t relate factors to overall effectiveness • Inputs • Raw materials • Measured against standards • Insufficient for overall assessment
#libdata4impact Principles for Applying Outcomes Assessment • Center on users • Assess changes in service/resources use • Relate to inputs - identify “best practices” • Use variety of methods to corroborate conclusions • Choose small number of outcomes • Need not address every aspect of service • Adopt continuous process
#libdata4impact What We Know About Assessment • Ongoing process to understand & improve initiatives • Librarians are busy with day-to-day work & assessment can become another burden • Can build on what your library has already done
Digital Library Manager, Worthington Libraries (OH) Kara Reuter Leveraging What You Know and Filling in the Gaps
#libdata4impact Anecdotes Casual observation Majority of one Fine for certain circumstances… Informal assessment “Suggestion Box” by John Pavelka, is licensed under CC BY 2.0
#libdata4impact Data driven Evidence based Accepted methods Recognized as rigorous Generalizable Formal assessment “Lego Alien DNA extractor diorama” by Paul Hartzog, is licensed under CC BY-SA 2.0
#libdata4impact Understand a phenomenon Build meaning Take a snapshot Tell a story Goal of assessment “building blocks” by Elisa Maser is licensed under CC BY 2.0
#libdata4impact 1. Define the problem 2. Environmental scanning 3. Prepare the plan 4. Collect data 5. Analyze data 6. Interpret and report Process “Steps” by Phil Whitehouse is licensed under CC BY 2.0 Valparaiso, Chile
#libdata4impact Carney, T. F. (1990). “The Ladder of Analytical Abstraction.” In Miles, M. B. & Huberman, A. M. (1994). Qualitative Data Analysis: An Expanded Sourcebook (2nd ed.). Thousand Oaks, CA: Sage. [p. 92] Process Ladder of abstraction
#libdata4impact Creswell, J. W. (1998). Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand Oaks, CA: Sage. [p. 143] Process Data analysis spiral
#libdata4impact User in their context Community perspective Ask the right question(s) Define the problem “FOCUS” by Iain Farrell is licensed under CC BY-ND 2.0 Sachuest Point National Wildlife Refuge, Rhode Island
#libdata4impact “Family Is Looking Up” by Troy Tolley is licensed under CC BY-ND 2.0 Peru, Indiana Define the problem User in context
#libdata4impact “Community “ by Rebecca Siegel is licensed under CC BY 2.0 Define the problem Community perspective
#libdata4impact Kara Reuter, There is such a thing as a stupid question https://www.someecards.com/usercards [now defunct] Define the problem Ask the right question(s)
#libdata4impact Existing data sources Other research Environmental scanning “newspaper reading” by balu is licensed under CC BY 2.0
#libdata4impact Existing data sources Environmental scanning
#libdata4impact Other research Environmental scanning • Pew Research Center • IMLS Public Libraries Survey • PLA Public Library Data Service • OCLC Reports • ACRL White Papers and Reports • NCES Library Statistics Program
#libdata4impact What kind of conclusion do you want to come to? How will you analyze the data? What data will you collect? What question(s) will you ask? Prepare the plan “End” by Eric Fischer is licensed under CC BY 2.0 San Francisco, California
#libdata4impact ”What went wrong??” by Jerry McFarland, Licensed under CC BY-NC 2.0 What could go wrong?
#libdata4impact What’s your vision for the library’s future? The question
#libdata4impact What’s your vision for the library’s future? The data
#libdata4impact Analysis What’s your vision for the library’s future? • Collection/materials • Technology • Facilities • Programming
#libdata4impact Conclusion What’s your vision for the library’s future? People envisioned more of what the library already does.
#libdata4impact “The Tool Box” by Tim Green is licensed under CC BY 2.0 Otley, West Yorkshire, England We can fix this!
#libdata4impact What’s your vision for the library’s future? One thing that would make my life better… The question
#libdata4impact What’s your vision for the library’s future? The data One thing that would make my life better…
#libdata4impact What’s your vision for the library’s future? Analysis • Collection/materials • Technology • Facilities • Programming One thing that would make my life better… • Freedom – Time, Money • Affection – Pets, Family • Food • Fantasy
#libdata4impact What’s your vision for the library’s future? Conclusion People envisioned more of what the library already does. One thing that would make my life better… People long for more freedom in their lives and seek warmth and affection in their relationships. They are both practically-minded, concerning themselves with food and diet, but also whimsical, wishing for magical or fantastical scenarios.
#libdata4impact Freedom Impacts and Outcomes Fantasy FoodAffection Photos by Worthington Libraries, used with permission
Questions and Discussion Kara Reuter Digital Library Manager, Worthington Libraries kreuter@worthingtonlibraries.org Thank you! Lynn Silipigni Connaway Senior Research Scientist and Director of User Research, OCLC Research connawal@oclc.org #libdata4impact
#libdata4impact Webinar Series: Evaluating and Sharing Your Library’s Impact Part 1: April 24 Kara Reuter User-centered Assessment: Leveraging What You Know and Filling in the Gaps Part 2: August 14 Linda Hofschire Digging into Assessment Data: Tips, Tricks, and Tools of the Trade Part 3: October 3 Melissa Bowles-Terry Take Action: Using and Presenting Research Findings to Make Your Case For more information: https://www.webjunction.org/news/ webjunction/webinar-series-research-assessment.html
Series Learner Guide Use alone or with others to apply what you’re learning between sessions. 13 pages of questions, activities, and resources. Customizable to meet your team’s needs!

More Related Content

User-centered assessment: Leveraging what you know and filling in the gaps.

  • 1. #libdata4impact Webinar Series: Evaluating and Sharing Your Library’s Impact Part 1: April 24 Kara Reuter User-centered Assessment: Leveraging What You Know and Filling in the Gaps Part 2: August 14 Linda Hofschire Digging into Assessment Data: Tips, Tricks, and Tools of the Trade Part 3: October 3 Melissa Bowles-Terry Take Action: Using and Presenting Research Findings to Make Your Case For more information: https://www.webjunction.org/news/ webjunction/webinar-series-research-assessment.html
  • 2. Series Learner Guide Use alone or with others to apply what you’re learning between sessions. 13 pages of questions, activities, and resources. Customizable to meet your team’s needs!
  • 3. • Research devoted exclusively to the challenges facing libraries and archives • Research Library Partnership includes working groups to collaborate with institutions on research and issues • Lifelong learning from WebJunction, for all library staff and volunteers • All connected through a global network of 16,000+ member libraries • Global and Regional Councils bring worldwide viewpoints together, informing and guiding the cooperative from their unique perspective.
  • 4. #libdata4impact Research Library Partnership: Library Assessment Interest Group • The OCLC Research Library Partnership invites librarians at partner institutions to participate in a new Library Assessment Interest Group, beginning in April 2018. • This interest group will accompany and extend the Webinar Series: Evaluating and Sharing Your Library's Impact
  • 5. Lynn Silipigni Connaway Senior Research Scientist and Director of User Research, OCLC Research connawal@oclc.org @LynnConnaway
  • 7. #libdata4impact Some Initial Questions 1. What is your definition of assessment? 2. What comes to mind when you hear the term “assessment”? 3. What benefits do you see for assessment? 4. What are your concerns?
  • 8. #libdata4impact Why Assessment? • Answers questions: • What do users/stakeholders want & need? • How can services/programs better meet needs? • Is what we do working? • Could we do better? • What are problem areas? • Traditional stats don’t tell whole story
  • 9. #libdata4impact Importance of Assessment “Librarians are increasingly called upon to document and articulate the value of academic and research libraries and their contribution to institutional mission and goals.” (ACRL Value of Academic Libraries, 2010, p. 6) Association of College and Research Libraries. (2010) Value of Academic Libraries: A Comprehensive Research Review and Report. Researched by Megan Oakleaf. Chicago: Association of College and Research Libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf
  • 10. #libdata4impact Steps in Assessment Process 1. Why? • Identify purpose 2. Who? • Identify team 3. How? • Choose model/approach/method 4. Commit! • Training/planning
  • 11. #libdata4impact Outcomes Assessment Basics Outcomes: “The ways in which library users are changed as a result of their contact with the library’s resources and programs.” (ALA/ACRL. 1998) “Libraries cannot demonstrate institutional value to maximum effect until they define outcomes of institutional relevance and then measure the degree to which they attain them.” (Kaufman & Watstein, 2008, p. 227) ALA/ACRL. (1998). Task Force on Academic Library Outcomes Assessment Report. Kaufman, P., & Watstein, S. B. (2008). Library Value (Return on Investment, ROI) and the Challenge of Placing a Value on Public Services. Reference Services Review, 36(3), 226-231.
  • 12. #libdata4impact Outputs & Inputs • Outputs • Quantify the work done • Don’t relate factors to overall effectiveness • Inputs • Raw materials • Measured against standards • Insufficient for overall assessment
  • 13. #libdata4impact Principles for Applying Outcomes Assessment • Center on users • Assess changes in service/resources use • Relate to inputs - identify “best practices” • Use variety of methods to corroborate conclusions • Choose small number of outcomes • Need not address every aspect of service • Adopt continuous process
  • 14. #libdata4impact What We Know About Assessment • Ongoing process to understand & improve initiatives • Librarians are busy with day-to-day work & assessment can become another burden • Can build on what your library has already done
  • 15. Digital Library Manager, Worthington Libraries (OH) Kara Reuter Leveraging What You Know and Filling in the Gaps
  • 16. #libdata4impact Anecdotes Casual observation Majority of one Fine for certain circumstances… Informal assessment “Suggestion Box” by John Pavelka, is licensed under CC BY 2.0
  • 17. #libdata4impact Data driven Evidence based Accepted methods Recognized as rigorous Generalizable Formal assessment “Lego Alien DNA extractor diorama” by Paul Hartzog, is licensed under CC BY-SA 2.0
  • 18. #libdata4impact Understand a phenomenon Build meaning Take a snapshot Tell a story Goal of assessment “building blocks” by Elisa Maser is licensed under CC BY 2.0
  • 19. #libdata4impact 1. Define the problem 2. Environmental scanning 3. Prepare the plan 4. Collect data 5. Analyze data 6. Interpret and report Process “Steps” by Phil Whitehouse is licensed under CC BY 2.0 Valparaiso, Chile
  • 20. #libdata4impact Carney, T. F. (1990). “The Ladder of Analytical Abstraction.” In Miles, M. B. & Huberman, A. M. (1994). Qualitative Data Analysis: An Expanded Sourcebook (2nd ed.). Thousand Oaks, CA: Sage. [p. 92] Process Ladder of abstraction
  • 21. #libdata4impact Creswell, J. W. (1998). Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand Oaks, CA: Sage. [p. 143] Process Data analysis spiral
  • 22. #libdata4impact User in their context Community perspective Ask the right question(s) Define the problem “FOCUS” by Iain Farrell is licensed under CC BY-ND 2.0 Sachuest Point National Wildlife Refuge, Rhode Island
  • 23. #libdata4impact “Family Is Looking Up” by Troy Tolley is licensed under CC BY-ND 2.0 Peru, Indiana Define the problem User in context
  • 24. #libdata4impact “Community “ by Rebecca Siegel is licensed under CC BY 2.0 Define the problem Community perspective
  • 25. #libdata4impact Kara Reuter, There is such a thing as a stupid question https://www.someecards.com/usercards [now defunct] Define the problem Ask the right question(s)
  • 26. #libdata4impact Existing data sources Other research Environmental scanning “newspaper reading” by balu is licensed under CC BY 2.0
  • 28. #libdata4impact Other research Environmental scanning • Pew Research Center • IMLS Public Libraries Survey • PLA Public Library Data Service • OCLC Reports • ACRL White Papers and Reports • NCES Library Statistics Program
  • 29. #libdata4impact What kind of conclusion do you want to come to? How will you analyze the data? What data will you collect? What question(s) will you ask? Prepare the plan “End” by Eric Fischer is licensed under CC BY 2.0 San Francisco, California
  • 30. #libdata4impact ”What went wrong??” by Jerry McFarland, Licensed under CC BY-NC 2.0 What could go wrong?
  • 31. #libdata4impact What’s your vision for the library’s future? The question
  • 32. #libdata4impact What’s your vision for the library’s future? The data
  • 33. #libdata4impact Analysis What’s your vision for the library’s future? • Collection/materials • Technology • Facilities • Programming
  • 34. #libdata4impact Conclusion What’s your vision for the library’s future? People envisioned more of what the library already does.
  • 35. #libdata4impact “The Tool Box” by Tim Green is licensed under CC BY 2.0 Otley, West Yorkshire, England We can fix this!
  • 36. #libdata4impact What’s your vision for the library’s future? One thing that would make my life better… The question
  • 37. #libdata4impact What’s your vision for the library’s future? The data One thing that would make my life better…
  • 38. #libdata4impact What’s your vision for the library’s future? Analysis • Collection/materials • Technology • Facilities • Programming One thing that would make my life better… • Freedom – Time, Money • Affection – Pets, Family • Food • Fantasy
  • 39. #libdata4impact What’s your vision for the library’s future? Conclusion People envisioned more of what the library already does. One thing that would make my life better… People long for more freedom in their lives and seek warmth and affection in their relationships. They are both practically-minded, concerning themselves with food and diet, but also whimsical, wishing for magical or fantastical scenarios.
  • 40. #libdata4impact Freedom Impacts and Outcomes Fantasy FoodAffection Photos by Worthington Libraries, used with permission
  • 41. Questions and Discussion Kara Reuter Digital Library Manager, Worthington Libraries kreuter@worthingtonlibraries.org Thank you! Lynn Silipigni Connaway Senior Research Scientist and Director of User Research, OCLC Research connawal@oclc.org #libdata4impact
  • 42. #libdata4impact Webinar Series: Evaluating and Sharing Your Library’s Impact Part 1: April 24 Kara Reuter User-centered Assessment: Leveraging What You Know and Filling in the Gaps Part 2: August 14 Linda Hofschire Digging into Assessment Data: Tips, Tricks, and Tools of the Trade Part 3: October 3 Melissa Bowles-Terry Take Action: Using and Presenting Research Findings to Make Your Case For more information: https://www.webjunction.org/news/ webjunction/webinar-series-research-assessment.html
  • 43. Series Learner Guide Use alone or with others to apply what you’re learning between sessions. 13 pages of questions, activities, and resources. Customizable to meet your team’s needs!