Technology Adoption Decisions for Mission-Critical Learning Infrastructure: Viewing Learning Management System Selection Issues through Three Different Lenses As any institution conducts an investigation to find an alternative LMS solution, different stakeholders ask different questions relating to the same concepts or issues. We have put these questions into the following matrix to help institutional stakeholders conduct a rigorous analysis. Issue Accessibility Contractual constraints Ease of use Use of commodity hardware Teaching and Learning Perspective Technology Management Perspective Organizational Administration Perspective Can instructors and students with disabilities use the LMS to accomplish all teaching and learning tasks? Do instructors and students have access to the necessary assistive technology tools and information to make the LMS useful? Will this LMS’s contract process be an obstacle to making a timely switch? Will the time it takes to resolve contract-related issues prevent immediate resolution of technical problems and impact teaching and learning negatively? Does the LMS work with all browsers and platforms? Is it feasible to modify the system to improve accessibility? Do campus labs have stations with the necessary assistive technology tools and information to make the LMS useful? Will the contract contain gaps that necessitate additional agreements for the vendor to perform various services? Will resolution of contract-related issues prevent immediate resolution of technical problems? How can the university prevent lawsuits related to unequal access to the system? What types of accessibility information must be disseminated to the instructors and students with disabilities? Is the LMS environment intuitive enough for instructors and students to complete tasks with little to no training? Can the environment be organized effectively? For example, can all online materials for a given topic be provided in one region of the online course, regardless of whether they are implemented using different LMS tools? Will the type of hardware limit the use of any types of materials for coursework (e.g., publisher-created materials)? Is the LMS written in a common language for which it will be easy to find qualified programmers? Is the LMS easy to install and maintain? Are there clauses in the contract to hold a vendor accountable for problems caused by the vendor? Does the contract period have opt-out points? Does the contract period align with the academic year? Are there benefits for joining consortium contracts within a university system? How accessible is the aggregated data that is collected? Is it easy to extract and summarize logs (e.g., usage levels) and other data for ROI analysis? Is the LMS capable of running on different hardware configurations, or must the university use proprietary hardware that requires specialized training for system administrator(s)? Will the university be able to choose cost-effective equipment to run the LMS? Will hardware issues present a barrier to future LMS changes, if commodity hardware is not a CC BY 4.0 - Joshua L. Mindel & Kevin Kelly 1 Technology Adoption Decisions for Mission-Critical Learning Infrastructure: Viewing Learning Management System Selection Issues through Three Different Lenses Issue Extensibility/ Customizability Features Integration with IT infrastructure Teaching and Learning Perspective Is it possible for instructors and students to suggest changes or new functionalities to help accomplish instructional goals? To what extent can the LMS be customized to suit the heterogeneous needs of the campus learning community? For example, can the online content in an engineering course be organized differently than the writing portfolios in an English writing course? Do the LMS features allow instructors and students to perform tasks as they would like, or must users modify tasks to fit system limitations? To what extent does the LMS provide a set of tools that enables instructors to establish a vibrant and effective online community outside the classroom? In addition to the rudimentary file upload and download capabilities, are there advanced community features such as forums, wikis, instant messaging, RSS feeds, etc.? Is the LMS integrated with other campus information systems and services? For example, are course shells automatically created each semester, or must instructors request them each time? Are students automatically enrolled in appropriate CC BY 4.0 - Joshua L. Mindel & Kevin Kelly Technology Management Perspective Can the hardware be integrated with existing IT infrastructure (e.g., racks in data center, power, connectivity)? Can our technology management staff make appropriate changes to hardware and software to meet educational needs expressed by instructors and students as requirements evolve? Will customizations of open source create a product that breaks too far away from the original, isolating the university from external support? Can a commercial vendor LMS be customized? How easy is it to develop and implement faculty and student training to use the technical features of the LMS? To what extent do the features meet or exceed requirements expressed by instructors? If features do not meet immediate requirements, is there sufficient capability to add on features to provide such functionality? To what extent is there support (vendor-based, community, etc.) to add functionality as requirements evolve? What level of effort is required to integrate the LMS with student management databases, campus authentication processes, campus portals, or library electronic reserves, and other academic technologies? What level of effort is required to implement Organizational Administration Perspective possibility? Will customizations of open source create a product that breaks too far away from the original, isolating the university from external support? In anticipation that campus needs will evolve over time, how much flexibility and freedom does the campus have in enhancing the LMS to serve unanticipated needs? Can any of the features be used to provide institution-level reporting for accreditation or for generating reports useful to administrators of the LMS and executive management in the university? Can the LMS be integrated with existing academic technologies? If so, can the campus perform these integrations in-house or does it need to hire outside help? Can information from the LMS be made available and other reporting tools used to administer 2 Technology Adoption Decisions for Mission-Critical Learning Infrastructure: Viewing Learning Management System Selection Issues through Three Different Lenses Issue Long-term viability Migration/ Transition Open Source Teaching and Learning Perspective course shells each semester? If not, can students enroll themselves, or will instructors have to enroll them? Must instructors and students use a separate log in for LMS? Can the system gradebook be used to submit final grades at end of semester? Will instructors and students soon have to learn how to use a new LMS again, or will they be able to stick with this system for a long enough period to benefit and learn from past experiences? In the event of a transition, do the instructors have to start over from scratch, or can they move materials and activities from one LMS to another? Should entire departments make the transition at the same time to provide a consistent experience for the students and to benefit from peer mentors? At a minimum, does the new LMS have the same set of features as the legacy system? Does the open source community share best practices in online teaching and learning with this LMS? CC BY 4.0 - Joshua L. Mindel & Kevin Kelly Technology Management Perspective Organizational Administration Perspective university policies (e.g., if a student drops a course)? Does the LMS leverage current and standards-based technology to ease integration today and in the future? the university? Can integration be used to implement university policies? Will the academic technology team soon have to lead the investigation process again because the LMS may disappear? Will the campus soon have to go through the investigation process again because the LMS product may disappear? Can the academic technology team adequately support faculty and students using two different LMS solutions during both testing and transition? How many people can the team train properly during a semester? How many can the team train properly during breaks? What types of self-help materials must the team make to assist those faculty who do not want to attend training? What are the technical requirements of converting the campus from the legacy LMS to the new system? Do we have the staff resources to handle this transition? What is the best way to interact with the open source community? If the campus does not have programmers, how does the open source "bounty" system work? Can the open source LMS be used "right out of the box" How long does the campus need to support two LMS solutions in parallel? What types of outreach need to be conducted? What rollout method should be used? How do the administrators respond to faculty who do not want to move to an alternative LMS? Can the transition phase be timed to end with the legacy LMS contract (if applicable), or does the campus need to extend the contract for a short period to finish the transition? What is the level of effort— and therefore cost—of transitioning to the new LMS? What are the benefits versus the costs of open source? If moving from a commercial LMS to an open source system, how should the campus reallocate the money saved from the legacy LMS licensing fees? 3 Technology Adoption Decisions for Mission-Critical Learning Infrastructure: Viewing Learning Management System Selection Issues through Three Different Lenses Issue Teaching and Learning Perspective Performance Does the LMS meet teaching and learning expectations? Is the system responsive when administering courses, or when students interact with courses? Quality Describe the quality of the teaching and learning experience. Can instructors and students perform tasks without needing to know about the technology? Do the LMS features work as expected? For example, does restricting access to a selected resource actually restrict access in all situations? Is the LMS highly available? Will online coursework in the LMS be available whenever the instructors or students need it? Can instructors and students trust the LMS to stay up and running through exams, or do they need to stay with paper-based exams? What are the risks of conducting a high-stakes exam in the LMS environment? Is the LMS service stable and is there sufficient documentation available? Reliability and Stability Risks (Strategic, Operational, Legal) CC BY 4.0 - Joshua L. Mindel & Kevin Kelly Technology Management Perspective (i.e., without much modification)? If the university is currently paying a vendor to host the legacy LMS, is it possible to pay a different group to host the open source alternative? How can the campus best take advantage of the control gained by using open source? Does the LMS meet technology capability expectations? Does the LMS application and underlying database perform satisfactorily given the number of users, data requirements, and underlying platform capabilities? Describe the quality of the technology management experience related to the LMS. Is the LMS relatively stable and secure? Will the LMS be up and available 99.9% or even 99.99% of the time? If the LMS is hosted on campus, how often should the academic technology team require maintenance windows? What are the most stable underlying platform and operating system? Can bug fixes be put into production during the semester, or does the academic technology team need to wait until breaks between semesters each time? To what extent has the LMS vendor or the user community Organizational Administration Perspective Does the LMS meet expectations related to university mission and strategic goals? If the system does not meet performance expectations, how will administration address vocalized concerns? Describe the quality of the administration experience related to the LMS. Does the product reflect the high standards of our organization? Will instructors and students feel empowered and pleased to use the LMS? Will the LMS be up and available 99.9% or even 99.99% of the time? If the LMS does not meet stability expectations, how will administration address vocalized concerns? Legal risks include copyright and licensing infringement as well as managing the risk of combining open source code with the company's own, internally-developed source code. Strategic risks include security risks 4 Technology Adoption Decisions for Mission-Critical Learning Infrastructure: Viewing Learning Management System Selection Issues through Three Different Lenses Issue Teaching and Learning Perspective Scalability Can large classes be conducted in the LMS as easily as small classes? Will scalability stresses cause instructors and students to experience blackouts or brownouts that impact teaching and learning? How quickly can scalability issues be resolved so teaching and learning can resume? Security Is instructor and student personal data secure during the log-in process? If students need to pay for extended learning courses over the Internet, is the credit card information secure? Is intellectual property secure, or will people be able to steal instructors' work? Is the instructor access protected so that students cannot access it? CC BY 4.0 - Joshua L. Mindel & Kevin Kelly Technology Management Perspective Organizational Administration Perspective identified security as a strategic interest and allocated resources to auditing and reporting vulnerabilities? such as software vulnerabilities and dangerous source code. To what extent has the LMS vendor or the user community identified security as a strategic interest and allocated resources to auditing and reporting vulnerabilities? Can the LMS technology and support organization scale to serve the needs of a campus of our size? Will the LMS software require additional costs for hardware to make it truly scalable? What number of users is a high enough number to assure us that the system is scalable to the entire population? Can the LMS technology and support organization scale to serve the needs of a campus of our size? Will large numbers of users overload the server processor(s)? Does the configuration require load balancing to handle to high demand? Do certain LMS features or functions fail with high-volume use? At what point should the application, data, and database layers be given their own server(s)? How can the academic technology team make the system scalable if adding more hardware is not an available option? What are the symptoms of inability to scale? Frequent system crashes? Slow system performance? Inadequate response to instructor and resource issues? Does the academic technology team need to work with the campus IT group to increase network security? How does the team create and renew Secure Sockets Layer (SSL) certificates, if the LMS is hosted on campus? What security questions does the team need to ask the vendor if the vendor will host the LMS? Is there adequate vendor or What efforts need to be coordinated among various centralized and decentralized technology staff (e.g., lab managers, college technology staff) to promote good security practices across the campus? Will this LMS help us comply with all suitable privacy and safeguard regulations? 5 Technology Adoption Decisions for Mission-Critical Learning Infrastructure: Viewing Learning Management System Selection Issues through Three Different Lenses Issue Teaching and Learning Perspective Staffing and support How quickly will someone respond to help requests by faculty and students? Total Cost of Ownership (TCO) Will TCO include financial incentives for faculty to get training, to set up an LMS environment for coursework, to use the LMS in an exemplary manner, or to be a peer mentor? How much effort will be required of new users in order to set up and regularly use the system? Technology Management Perspective community support to rapidly report and address security vulnerabilities? How close can the campus get to 24/7 support for the LMS? What contingency plans are in place if the server goes down when no one is on campus? Do we have the staff to support this LMS? Can we train users on this LMS with our existing staff, or will we need to acquire additional support staff? What outside resources are available to help when our in-house team requires assistance? Which aspects of maintaining the LMS and providing training and support could be automated or partially automated to help keep TCO reasonable? What about enhancements? Organizational Administration Perspective Can the university competently support the LMS? Is there vendor or community support to which staff can turn when they need additional resources? Is there budget to leverage outside support? How will an open source LMS be supported since there is no single vendor “standing behind” the product? Does this LMS help us achieve our mission as an educational institution? Where does the LMS fall in the priority list of all budget items? Which costs within the LMS TCO can be reduced without impacting product or services dramatically? This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0), and may be adapted or modified to meet different or additional institutional needs. The decision-making matrix is a companion to the following book chapter: Mindel, J.L., & Kelly, K. (2011). Stakeholder issues for adopting a learning management system. In S. Hirtz and K. Kelly (Eds.), Education for a Digital World 2.0: Innovations in Education. Victoria, BC: Crown Publications. NOTE: The digital version of the companion book chapter also is licensed as a Creative Commons work (CC BY-NC-ND). Any part may be reproduced without permission but with attribution to the authors. CC BY 4.0 - Joshua L. Mindel & Kevin Kelly 6