Unlocking the Black Box: Exploring the use of Generative AI (ChatGPT) in Information Systems Research
Authors: 
Rohani Rohan, Lawal Ibrahim Dutsinma Faruk, Kittiphan Puapholthep, Debajyoti PalAuthors Info & Claims
Rohani Rohan, Lawal Ibrahim Dutsinma Faruk, Kittiphan Puapholthep, Debajyoti PalAuthors Info & ClaimsArticle No.: 17, Pages 1 - 9
Abstract
With the gaining popularity of generative AI tools like ChatGPT and their usage across several domains and disciplines, the question that naturally arises is how it can help the Information Systems (IS) researchers? Measuring hidden or latent constructs is one critical and primitive aspects of the IS domain that has always been challenging due to its abstractness. How good or bad these specially trained AI-based models are with respect to their conceptual understanding capabilities of specific IS constructs together with their usage for the purpose of testing IS theories is an unknown area. We set out to explore these unknown aspects in this work by conducting two separate experiments with ChatGPT using the already proven and robust Technology Acceptance Model (TAM) as the reference. Our results suggest that ChatGPT has good conceptual understanding of the presented latent constructs, although there might be certain validity issues in case of complex models. Therefore, it shows promise in the broader aspect of testing theories, but not without its limitations that we present in this research.
References
[1]
Davis, F.D. 1986. A TECHNOLOGY ACCEPTANCE MODEL FOR EMPIRICALLY TESTING NEW END-USER INFORMATION SYSTEMS: THEORY AND RESULTS.
[2]
Dutsinma, F.L.I. 2022. Personality is to a Conversational Agent What Perfume is to a Flower. IEEE Consumer Electronics Magazine. (2022), 1–1.
[3]
Dwivedi, Y.K. 2023. Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management. 71, March (Aug. 2023), 102642.
[4]
Hair, J.F. 2012. The Use of Partial Least Squares Structural Equation Modeling in Strategic Management Research: A Review of Past Practices and Recommendations for Future Applications. Long Range Planning. 45, 5 (2012), 320–340.
[5]
Hair, J.F. 2019. When to use and how to report the results of PLS-SEM. European Business Review. 31, 1 (Jan. 2019), 2–24.
[6]
Joe, H. 2017. An updated and expanded assessment of PLS-SEM in information systems research. Industrial Management & Data Systems. 117, 3 (Jan. 2017), 442–458.
[7]
Johnson, M.S. 2022. Psychometric Methods to Evaluate Measurement and Algorithmic Bias in Automated Scoring. Journal of Educational Measurement. 59, 3 (Sep. 2022), 338–361.
[8]
Lu, L. 2022. Measuring consumer-perceived humanness of online organizational agents. Computers in Human Behavior. 128, (Mar. 2022), 107092.
[9]
Ma, X. and Huo, Y. 2023. Are users willing to embrace ChatGPT? Exploring the factors on the acceptance of chatbots from the perspective of AIDUA framework. Technology in Society. 75, (Nov. 2023), 102362.
[10]
Marangunić, N. and Granić, A. 2015. Technology acceptance model: a literature review from 1986 to 2013. Universal Access in the Information Society. 14, 1 (Mar. 2015), 81–95.
[11]
Najafali, D. 2023. Truth or Lies? The Pitfalls and Limitations of ChatGPT in Systematic Review Creation. Aesthetic Surgery Journal. 43, 8 (Jul. 2023), NP654–NP655.
[12]
Niu, B. and Mvondo, G.F.N. 2024. I Am ChatGPT, the ultimate AI Chatbot! Investigating the determinants of users’ loyalty and ethical usage concerns of ChatGPT. Journal of Retailing and Consumer Services. 76, (Jan. 2024), 103562.
[13]
Ooi, K. 2023. The Potential of Generative Artificial Intelligence Across Disciplines: Perspectives and Future Directions. Journal of Computer Information Systems. 00, 00 (Oct. 2023), 1–32.
[14]
Pal, D. and Patra, S. 2021. University Students’ Perception of Video-Based Learning in Times of COVID-19: A TAM/TTF Perspective. International Journal of Human–Computer Interaction. 37, 10 (Jun. 2021), 903–921.
[15]
Rohan, R. 2023. A systematic literature review of cybersecurity scales assessing information security awareness. Heliyon. 9, 3 (Mar. 2023), e14234.
[16]
Spatola, N. 2021. Perception and Evaluation in Human–Robot Interaction: The Human–Robot Interaction Evaluation Scale (HRIES)—A Multicomponent Approach of Anthropomorphism. International Journal of Social Robotics. 13, 7 (Nov. 2021), 1517–1539.
[17]
Surameery, N.M.S. and Shakor, M.Y. 2023. Use Chat GPT to Solve Programming Bugs. International Journal of Information technology and Computer Engineering. 3, 31 (Jan. 2023), 17–22.
[18]
Tiwari, C.K. 2023. What drives students toward ChatGPT? An investigation of the factors influencing adoption and usage of ChatGPT. Interactive Technology and Smart Education. ahead-of-p, ahead-of-print (Aug. 2023).
[19]
Venkatesh 2012. Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly. 36, 1 (Apr. 2012), 157.
[20]
Worthington, R.L. and Whittaker, T.A. 2006. Scale Development Research. The Counseling Psychologist. 34, 6 (Nov. 2006), 806–838.
[21]
Yousafzai, S.Y. 2007. Technology acceptance: a meta‐analysis of the TAM: Part 1. Journal of Modelling in Management. 2, 3 (Nov. 2007), 251–280.
[22]
Yousafzai, S.Y. 2007. Technology acceptance: a meta‐analysis of the TAM: Part 2. Journal of Modelling in Management. 2, 3 (Nov. 2007), 281–304.
Index Terms
- Unlocking the Black Box: Exploring the use of Generative AI (ChatGPT) in Information Systems Research
Recommendations
Specifying formative constructs in information systems research
While researchers go to great lengths to justify and prove theoretical links between constructs, the relationship between measurement items and constructs is often ignored. By default, the relationship between construct and item is assumed to be ...
Analyzing information systems researchers' productivity and impacts: A perspective on the H index
Quantitative assessments of researchers' productivity and impacts are crucial for the information systems (IS) discipline. Motivated by its growing popularity and expanding use, we offer a perspective on the h index, which refers to the number of papers ...
Two decades of information systems: a bibliometric review
Information systems (IS) is a vital research area in both science and engineering, whose revolutions in terms of theories, techniques, and applications promote the evolution of human society. At the same time, the complexity and dynamics of IS raise the ...
Comments
Information & Contributors
Information
Published In
December 2023
303 pages
ISBN:9798400708497
DOI:10.1145/3628454
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 December 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
IAIT 2023
IAIT 2023: 13th International Conference on Advances in Information Technology
December 6 - 9, 2023
Bangkok, Thailand
Acceptance Rates
Overall Acceptance Rate 20 of 47 submissions, 43%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 836Total Downloads
- Downloads (Last 12 months)836
- Downloads (Last 6 weeks)93
Reflects downloads up to 10 Oct 2024
Other Metrics
Citations
Cited By
View allView Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in