Exploring Subjective and Objective Measures of System Effectiveness
Article No.: 149, Pages 1 - 8
Abstract
The effectiveness of information systems (IS) is essential for evaluating user performance between their behavior and computer. However, the effectiveness evaluation poses challenges due to the multidimensional nature of measures, especially in evaluating business intelligence systems (BI). This study aims to investigate measures from both subjective and objective of BI effectiveness and classify BI effectiveness into five dimensions based on the IS maturity model, including Operate, Consolidate, Integrate, Optimize, and Innovate. The study compares results obtained through self-reported with computer-recorded measures to gain a comprehensive understanding of system utilization. This research received 1480 questionnaires, accompanied by the collection of 1-year log data reflecting the actual system usage of each respondent. The findings reveal a partial alignment between self-reported and actual usage frequency depending on divergent effectiveness levels. These insights contribute to the complex dynamics underlying user behavior and provide practical implications for organizations seeking to enhance effective BI utilization.
Supplemental Material
- Download
- 61.26 MB
- Transcript
References
[1]
NoorUl Ain, Giovanni Vaia, William H. DeLone, and Mehwish Waheed. 2019. Two decades of research on business intelligence system adoption, utilization and success – A systematic literature review. Decision Support Systems 125, (October 2019), 113113. https://doi.org/10.1016/j.dss.2019.113113
[2]
Cigdem Altin Gumussoy, Aycan Pekpazar, Mustafa Esengun, Ayse Elvan Bayraktaroglu, and Gokhan Ince. 2022. Usability Evaluation of TV Interfaces: Subjective Evaluation Vs. Objective Evaluation. International Journal of Human–Computer Interaction 38, 7 (April 2022), 661–679. https://doi.org/10.1080/10447318.2021.1960093
[3]
Alejandrina Aranda, Oscar Dieste, and Natalia Juristo. 2014. Evidence of the presence of bias in subjective metrics: analysis within a family of experiments. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE ’14), May 13, 2014, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1–4. https://doi.org/10.1145/2601248.2601291
[4]
Tim Barnett, Franz W. Kellermanns, Allison W. Pearson, and Rodney A. Pearson. 2006. Measuring Information System Usage: Replication and Extensions. Journal of Computer Information Systems 47, 2 (December 2006), 76–85. https://doi.org/10.1080/08874417.2007.11645956
[5]
Jack J. Baroudi, Margrethe H. Olson, and Blake Ives. 1986. An empirical study of the impact of user involvement on system usage and information satisfaction. Commun. ACM 29, 3 (March 1986), 232–238. https://doi.org/10.1145/5666.5669
[6]
Scott A. Beal and James L. Eubanks. 2003. Self-Report Bias and Accuracy in a Simulated Work Setting. Journal of Organizational Behavior Management (March 2003). https://doi.org/10.1300/J075v22n01_02
[7]
Waranpong Boonsiritomachai, G. Michael McGrath, and Stephen Burgess. 2016. Exploring business intelligence and its depth of maturity in Thai SMEs. Cogent Business & Management 3, 1 (December 2016), 1220663. https://doi.org/10.1080/23311975.2016.1220663
[8]
Jim Davis, Gloria J. Miller, and Allan Russell. 2006. Information Revolution: Using the Information Evolution Model to Grow Your Business. John Wiley & Sons.
[9]
William H. DeLone and Ephraim R. McLean. 1992. Information Systems Success: The Quest for the Dependent Variable. Information Systems Research 3, 1 (March 1992), 60–95. https://doi.org/10.1287/isre.3.1.60
[10]
William H. DeLone and Ephraim R. McLean. 2016. Information Systems Success Measurement. ISY 2, 1 (August 2016), 1–116. https://doi.org/10.1561/2900000005
[11]
Kristens Gudfinnsson and Mattias Strand. 2017. Challenges with BI adoption in SMEs. In 2017 8th International Conference on Information, Intelligence, Systems & Applications (IISA) (August 2017), 1–6. https://doi.org/10.1109/IISA.2017.8316407
[12]
Scott Hamilton and Norman L. Chervany. 1981. Evaluating Information System Effectiveness - Part I: Comparing Evaluation Approaches. MIS Quarterly 5, 3 (1981), 55–69. https://doi.org/10.2307/249291
[13]
Matthew D. Haydock, Anubhav Mittal, Carissa F. Wilkes, David H. Lim, Elizabeth Broadbent, and John A. Windsor. 2019. Interaction between Objective Performance Measures and Subjective User Perceptions in the Evaluation of Medical Devices: A Case Study—ADDENDUM. International Journal of Technology Assessment in Health Care 35, 4 (January 2019), 361–361. https://doi.org/10.1017/S0266462319000564
[14]
Peter Hofmann, Caroline Samp, and Nils Urbach. 2020. Robotic process automation. Electron Markets 30, 1 (March 2020), 99–106. https://doi.org/10.1007/s12525-019-00365-8
[15]
Frederick A. Muckler and Sally A. Seven. 1992. Selecting Performance Measures: “Objective” versus “Subjective” Measurement. Hum Factors 34, 4 (August 1992), 441–455. https://doi.org/10.1177/001872089203400406
[16]
Celina M. Olszak. 2016. Toward Better Understanding and Use of Business Intelligence in Organizations. Information Systems Management 33, 2 (April 2016), 105–123. https://doi.org/10.1080/10580530.2016.1155946
[17]
Danique E. Paping, Jantien L. Vroegop, Simone P. C. Koenraads, Carlijn M.P. le Clercq, André Goedegebure, Robert J. Baatenburg de Jong, and Marc P. van der Schroeff. 2021. A smartphone application to objectively monitor music listening habits in adolescents. Journal of Otolaryngology - Head & Neck Surgery 50, 1 (February 2021), 11. https://doi.org/10.1186/s40463-020-00488-5
[18]
Stacie Petter, William DeLone, and Ephraim McLean. 2008. Measuring information systems success: models, dimensions, measures, and interrelationships. Eur J Inf Syst 17, 3 (June 2008), 236–263. https://doi.org/10.1057/ejis.2008.15
[19]
Aleš Popovič, Ray Hackney, Pedro Simões Coelho, and Jurij Jaklič. 2012. Towards business intelligence systems success: Effects of maturity and culture on analytical decision making. Decision Support Systems 54, 1 (December 2012), 729–739. https://doi.org/10.1016/j.dss.2012.08.017
[20]
Mark de Reuver and Harry Bouwman. 2015. Dealing with self-report bias in mobile Internet acceptance and usage studies. Information & Management 52, 3 (April 2015), 287–294. https://doi.org/10.1016/j.im.2014.12.002
[21]
Peter B. Seddon, Sandy Staples, Ravi Patnayakuni, and Matthew Bowtell. 1999. Dimensions of Information Systems Success. CAIS 2, (1999). https://doi.org/10.17705/1CAIS.00220
[22]
Detmar Straub, Moez Limayem, and Elena Karahanna-Evaristo. 1995. Measuring System Usage: Implications for IS Theory Testing. Management Science 41, 8 (August 1995), 1328–1342. https://doi.org/10.1287/mnsc.41.8.1328
[23]
Dalia Suša Vugec, Vesna Bosilj Vukšić, Mirjana Pejić Bach, Jurij Jaklič, and Mojca Indihar Štemberger. 2020. Business intelligence and organizational performance: The role of alignment with business process management. Business Process Management Journal 26, 6 (January 2020), 1709–1730. https://doi.org/10.1108/BPMJ-08-2019-0342
[24]
Bernadette Szajna. 1993. Determining information system usage: Some issues and examples. Information & Management 25, 3 (September 1993), 147–154. https://doi.org/10.1016/0378-7206(93)90037-T
[25]
G Torkzadeh and W. J Doll. 1999. The development of a tool for measuring the perceived impact of information technology on work. Omega 27, 3 (June 1999), 327–339. https://doi.org/10.1016/S0305-0483(98)00049-8
[26]
Noor Ul-Ain, Giovanni Vaia, and William DeLone. 2019. Business Intelligence System Adoption, Utilization and Success - A Systematic Literature Review. In Proceedings of the 52nd Hawaii International Conference on System Sciences (2019), 5888-5897. http://hdl.handle.net/10125/60024
[27]
Nils Urbach, Stefan Smolnik, and Gerold Riempp. 2009. The State of Research on Information Systems Success. Bus. Inf. Syst. Eng. 1, 4 (August 2009), 315–325. https://doi.org/10.1007/s12599-009-0059-y
[28]
Viswanath Venkatesh, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly 27, 3 (2003), 425–478. https://doi.org/10.2307/30036540
[29]
In Hye Yook, So Jin Park, Mun Joo Choi, Dai-Jin Kim, and In Young Choi. 2019. Factors Affecting Smartphone Usage Self-Report Levels. Stud Health Technol Inform 264, (August 2019), 1937–1938. https://doi.org/10.3233/shti190722
[30]
Jinhua Zhao. 2011. Subjective Measure of Car Dependence. Transportation Research Record 2231, 1 (January 2011), 44–52. https://doi.org/10.3141/2231-06
Index Terms
- Exploring Subjective and Objective Measures of System Effectiveness
Recommendations
Comparing Objective Measures and Perceptions of Cognitive Learning in an ERP Simulation Game: A Research Note
Enterprise Resource Planning (ERP) systems have had a significant impact on business organizations. These large systems offer opportunities for companies regarding the integration and functionality of information technology systems; in effect, companies ...
Objective vs. Subjective Measures of Error-Proneness for Rule-Based Programs
This paper relates four objective measures of program structure and three subjective ratings of program complexity to the number of postrelease documented errors contained within 80 commercially developed Prolog programs. All seven measures show a ...
Comments
Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Qualifiers
- Work in progress
- Research
- Refereed limited
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 137Total Downloads
- Downloads (Last 12 months)137
- Downloads (Last 6 weeks)35
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format