article

Evaluating information systems effectiveness - part i:: comparing evaluation approaches

Authors:

Scott Hamilton,

Norman L. ChervanyAuthors Info & Claims

MIS Quarterly, Volume 5, Issue 3

Pages 55 - 69

https://doi.org/10.2307/249291

Published: 01 September 1981 Publication History

Abstract

While the sue and costs of Management Information Systems (MIS) have become highly visible, little attention has been paid to assessing and communicating system effectiveness. Evaluation of system effectiveness is difficult due to its multidimensionality, its qualitative and quantitative aspects, and the multiple, and often conflicting, evaluator viewpoints.

This article provides an overview of what system effectiveness means and how it should be measured. It is the first of two articles to appear in consecutive issues of the MIS Quarterly. Starting with a definition of system effectiveness, this article examines evaluation of system effectiveness in terms of a conceptual hierarchy of system objectives. The hierarchy is used to discuss problems in, and recommendations for, evaluating system effectiveness, and to compare MIS evaluation approaches. The second article characterizes and compares the evaluator viewpoints on system effectiveness for decision makers in several functional groups involved in MIS implementation ᾢ user, MIS, internal audit, and management. The second article recommends several MIS evaluation approaches for incorporating multiple dimensions and multiple evaluator viewpoints into evaluations of information system effectiveness.

References

[1]

Alter, S.L. "A Study of Computers Aided Decision Making in Organizations," Unpublished Ph.D. Dissertation, MIT, Cambridge, Massachusetts, 1975.

[2]

Anderson, J.C., Schroeder, R.G., Tupy, S.E., and White, E.M. "MRP: A Study of Implementation and Practice," APICS Monograph, forthcoming 1981.

[3]

Appleton, D. "A Manufacturing Systems Cookbook: Part 2," Datamation, Volume 25, Number 3, June 1979, pp. 132-140.

[4]

Campbell, J.P. "On the Nature of Organization Effectiveness," New Perspectives on Organizational Effectiveness, J. P. Campbell, P. Goodman, and J.M. Pennings, eds., Josey-Bass, Boston, Massachusetts, 1977, pp. 13-55.

[5]

Carlson, E.O. "Evaluating the Impact of Information Systems," Management Informatics, Volume 3, March 1974, pp.57-67.

[6]

Cheney, H. "Measuring MIS Project Success," Proceedings of 9th Annual AIDS Conference, Chicago, Illinois, October 25-27, 1977, pp. 141-143.

[7]

Dickson, G.W. and Powers, R.F. "MIS Project Management: Myths, Opinions, and Reality," California Management Review, Volume 7, Spring 1973, pp. 147-156.

[8]

Dolotto, T.A., Bernstein, M.I., Dickson, R.S., Jr., France, N.A., Rosenblatt, B.A., Smith, D.M., and Steel, T.B., Jr. Data Processing in 1980-1985, John Wiley and Sons, New York, New York, 1976.

[9]

Ein-dor, P. and Segev, E. "Strategic Planning for Management Information Systems," Management Science, Volume 24, Number 15, November 1978, pp.1631-1641.

Digital Library

[10]

Gallagher, C.A. "Perceptions of the Value of a Management Science Implementation," Academy of Management Journal, Volume 1 7, Number 1, March 1974, pp. 46-55.

[11]

Ginzberg, M.J. "Improving MIS Project Selection," Research Paper #135A, Graduate School of Business, Columbia University, New York, New York, August 1978.

[12]

Greenberg, H.D., Stewart, R.J.S., Hanes, L.F., Kriebel, C.H., and Debons, A. "Productivity Measurement Systems for Administrative Services: Computing and Information Services," Proceedings of the Grantee's Conference on Research on Productivity Measurement Systems for Administrative Services, Tempe, Arizona, November 15-16, 1976.

[13]

Halbrecht, H.Z. Interview with R.E. McDonald, MIS Quarterly, Volume 1, Number 2, June 1977, pp. 7-11.

[14]

Hamilton, S. "A Survey of Data Processing Post Installation Evaluation Practices," MISRC WP# 80-06, University of Minnesota, Minneapolis, Minnesota, February 1980.

[15]

Keen, P.G.W. and Scott Morton, M.S. Decision Support Systems: An Organizational Perspective, Addison Wesley, Reading, Massachusetts, 1978.

[16]

Kriebel, C.H., Raviv, A., and Zia, H. "An Economics Approach to Modeling the Productivity of Information Systems," Technical Report NSP APR75- 20546/76/TR2R, Carnegie-Mellon University, Pittsburgh, Pennsylvania, July 1977.

[17]

Langefores, B. "Discussion of 'Determining Management Information Needs: A Comparison of Methods,'" MIS Quarterly, Volume 1, Number 4, December 1977.

[18]

Lientz, B.P., Swanson, E.B., and Tompkins, G.E. "Characteristics of Application Software Maintenance," Communications of the ACM, Volume 21, Number 6, June 1978, pp. 466-471.

Digital Library

[19]

Lucas, H.C., Jr. "Performance and Use of an Information System," Management Science, Volume 21, Number 4, April 1975, pp. 908-919.

Digital Library

[20]

McLean, E.R. and Soden, J. Strategic Planning for MIS, Wiley and Son, New York, New York, 1977.

[21]

Meals, D.W. "Systems Evaluation," Journal of Systems Management, Volume 16, Number 7, July 1977, pp. 6-9.

[22]

Molnar, J.J. and Rogers, D.L. "Organizational Effectiveness: An Empirical Comparison of the Goal and System Resource Approaches," The Sociological Quarterly, Volume 17, Number 2, Summer 1976, pp. 401-413.

[23]

Norton, D.P. and Rau, K.G. "A Guide to EDP Performance Management," QED Information Sciences, Wesley, Massachusetts, 1978.

[24]

O'Brien, J.F. "Methodology for Assessing the Impact of Computing and Information Systems on Users," Technical Report APR- 20546/77/TR6, Westinghouse R&D Center, Pittsburgh, Pennsylvania, August 1977.

[25]

Pearson, S.W. "Measurement of Computer User Satisfaction," Unpublished Ph.D. Dissertation, Arizona State University, Tempe, Arizona, 1977.

[26]

Price, J.L. "The Study of Organizational Effectiveness," The Sociological Quarterly, Volume 13. Number 4, Winter 1972, pp. 3-15.

[27]

Richardson, P.R. and Gordon, J.R.M. "Measuring Total Manufacturing Performance." Sloan Management Review, Volume 9, Number 2, Winter 1980, pp.47-58.

[28]

Schriven, M. "The Methodology of Evaluation: Formative and Summative Evaluation," Evaluating Action Programs, C.H. Weiss, ed., Allyn and Bacon, Boston, Massachusetts, 1972.

[29]

Seashore, S.E. and Yuchtman, E. "Factor Analysis of Organizational Performance," Administrative Science Quarterly, Volume 12, Number 2, December 1967, pp. 377-395.

[30]

Seward, H.H. "Evaluating Information Systems," The Information Systems Handbook, F. McFarland and R.L. Nolan, eds., Dow Jones Irwin, Homewood, Illinois, 1975, pp. 132-153.

[31]

Smith, G.L., Jr. "Participatory Approaches to the Development of Productivity Measures," Proceedings of the 9th Annual SMIS Conference, September 1977, pp 101-110.

[32]

Stabell, C.B. "Individual Differences in Management Decision Making Processes: A Study of Conversational Computer Usage," Ph.D. Dissertation, MIT, Cambridge, Massachusetts, 1974.

[33]

Varanelli, A. "An Organizational Approach to the Evaluation of Computing Center Effectiveness," Unpublished Paper, Department of Management, Pace University, New York, New York, 1978.

Cited By

Chen SChiu KHsu P(2024)Exploring Subjective and Objective Measures of System EffectivenessExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650963(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650963
Lee SLi NKim J(2024)Conceptual model for Mexican teachers' adoption of learning analytics systems: The integration of the information system success model and the technology acceptance modelEducation and Information Technologies10.1007/s10639-023-12371-729:11(13387-13412)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s10639-023-12371-7
Abernethy MDekker HGrafton J(2021)The Influence of Performance Measurement on the Processual Dynamics of Strategic ChangeManagement Science10.1287/mnsc.2019.344267:1(640-659)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1287/mnsc.2019.3442
Show More Cited By

Index Terms

Evaluating information systems effectiveness - part i:: comparing evaluation approaches

Recommendations

Evaluating information systems effectiveness - part ii:: comparing evaluator viewpoints

Evaluations of Management Information Systems (MIS) tend to be subjective and are influenced by the perceptions of system objectives, as well as the experiences with system performance in accomplishing organizational objectives. Consequently, the ...
Research of Evaluating Method To MIS
IFITA '09: Proceedings of the 2009 International Forum on Information Technology and Applications - Volume 01

A synthesized Evaluating method to MIS is brought forward in this paper, which is based on Delphi, AHP and fuzzy comprehensive evaluation, and put a coefficient to professors initiately during the evaluating process. It could to solve evaluating ...
Evaluating the Information Behaviour methods: Formative evaluations of two methods for assessing the functionality and usability of electronic information resources

The importance of user-centred evaluation is stressed by HCI academics and practitioners alike. However, there have been few recent evaluation studies of User Evaluation Methods (UEMs), especially those with the aim of improving methods rather than ...

Comments

Information & Contributors

Information

Published In

cover image MIS Quarterly

MIS Quarterly Volume 5, Issue 3

September 1981

71 pages

ISSN:0276-7783

Issue’s Table of Contents

Publisher

Society for Information Management and The Management Information Systems Research Center

United States

Publication History

Published: 01 September 1981

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
8
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen SChiu KHsu P(2024)Exploring Subjective and Objective Measures of System EffectivenessExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650963(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650963
Lee SLi NKim J(2024)Conceptual model for Mexican teachers' adoption of learning analytics systems: The integration of the information system success model and the technology acceptance modelEducation and Information Technologies10.1007/s10639-023-12371-729:11(13387-13412)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s10639-023-12371-7
Abernethy MDekker HGrafton J(2021)The Influence of Performance Measurement on the Processual Dynamics of Strategic ChangeManagement Science10.1287/mnsc.2019.344267:1(640-659)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1287/mnsc.2019.3442
Oh JLee ULee K(2019)Usability Evaluation Model for Biometric System considering Privacy Concern Based on MCDM ModelSecurity and Communication Networks10.1155/2019/87152642019Online publication date: 27-Mar-2019
https://dl.acm.org/doi/10.1155/2019/8715264
Bouaynaya W(2019)Characterization of Cloud Computing Reversibility as Explored by the DELPHI MethodInformation Systems Frontiers10.1007/s10796-019-09947-522:6(1505-1518)Online publication date: 27-Aug-2019
https://dl.acm.org/doi/10.1007/s10796-019-09947-5
Trutnev DVidiasova L(2018)Evaluation of Government Information Systems Effectiveness: The Case of e-Participation Portals in RussiaElectronic Participation10.1007/978-3-319-98578-7_3(26-38)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1007/978-3-319-98578-7_3
Al-Nassar B(2016)The influence of service quality in information system success model as an antecedent of mobile learning in education organisationsInternational Journal of Mobile Learning and Organisation10.1504/IJMLO.2017.08089311:1(41-62)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1504/IJMLO.2017.080893
Hou LXue LBui SKettinger W(2016)System sourcing and information processing capability in supply chainsInformation Technology and Management10.1007/s10799-016-0255-717:4(379-391)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1007/s10799-016-0255-7
Hausawi YAllen W(2015)Usable-Security EvaluationProceedings of the Third International Conference on Human Aspects of Information Security, Privacy, and Trust - Volume 919010.1007/978-3-319-20376-8_30(335-346)Online publication date: 2-Aug-2015
https://dl.acm.org/doi/10.1007/978-3-319-20376-8_30
Wu CHsu C(2015)How to Improve E-Satisfaction and E-Loyalty and Strengthen the Links Between ThemHuman Factors in Ergonomics & Manufacturing10.1002/hfm.2054925:3(353-369)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1002/hfm.20549
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents