38 THE NEW CHALLENGE OF BUSINESS VALUE: Time to Link Project Management Performance with Adoption Research? Chris Sauer Said Business School, Oxford University Oxford, United Kingdom Blaize Horner Reich Andrew Gemino Simon Fraser University Burnaby, Canada 1 INTRODUCTION When Working Group 8.6 was formed, there was only partial recognition among academics and practitioners that implementation of information technology did not automatically translate into adoption and diffusion. Rigorous study of these issues was thus well-motivated. In the last decade, focus on adoption has become mainstream for practice. For example, the UK's National Health Service Connecting for Health program has contractually required its suppliers not only to implement new medical record and booking systems, but also to secure their adoption. Today, organizations increasingly recognize the need to manage IT project investments to achieve business value. Project managers are tasked with more than just delivering an implemented or adopted IT system. According to our informants, delivery requires process change, organizational redesign, and benefits capture. Thus, in the National Health Service example, value will only be achieved when hospitals and primary care trusts deliver better health and service outcomes because they have restructured, adopted new processes, and managed for the value outcomes that the techPlease use the following format when citing this chapter: Sauer, C, Reich, B. H., and Gemino, A., 2007, in IFIP International Federation for Information Processing, Volume 235, Organizational Dynamics of Technology-Based Innovation: Diversifying the Research Agenda, eds. McMaster, T., Wastell, D., Femeley, E., and DeGross, J. (Boston: Springer), pp. 497-502. Part 7: Position Papers 498 nology makes possible. Put crudely, if the objective is to save costs by reducing headcount in a business unit, the value is only achieved when the relevant number of employees has left the building. One consequence of these changes is that the study of adoption and diffusion of information technology remains useful because they are a necessary prerequisite to securing value. However, they are no longer sufficient. We should be trying also to understand the point where the real interest lies, vis. value delivery and its antecedents. Our reason for making this observation is that we believe there would be mutual benefit from connecting the WG 8.6 agenda with our own area of research—project management performance—against which similar criticisms can be leveled. In this area, research has focused principally on understanding performance as a function of risk and project management practice (Figure 1). Performance has been most commonly construed as process performance (i.e., delivery against budget and schedule). Less commonly it has been construed as the extent to which the product delivered was of the scope and quality required. Only occasionally have researchers collected data about benefit achievement by the business. (And, curiously, almost never is adoption considered as a relevant dependent variable!) Our own research has sought to improve the accuracy and explanatory power of this kind of model by developing it into the interactive model shown in Figure 2. Changes we have made include Separating risks according to the time they occur in a project. A priori risks are known at project kickoff time and emergent risks occur during the life of the project. Employing more sophisticated data collection and data analysis techniques to explore the fiill range of performance and to investigate the interaction between the antecedents of project performance. Categorizing risks into risks and resources. We hypothesized that risks (i.e., size/ complexity and project volatility) and resources (i.e., knowledge resources and organizational support) would differentially interact with project management practices. Project Risk Requirements Uncertainty Technical Complexity Lack of Team Knowledge Lack of PM Knowledge Lack of Management Support Lack of User Participation Project Budget Project Duration Project Effort Project Management Practices Administrative Coordination Integration Expertise Coordination Figure 1. Project Risk Mediated by Project Management Practices Saueret al./The New Challenge of Business Value A Priori Risks and Resources Emergent Risks, Resources, and Actions 499 Project Performance Figure 2. An Interactive Model of IT Project Performance Adding a knowledge resources construct. This was suggested by our knowledge management research (Reich 2007; Reich and Wee 2006). It represents the knowledge and experience (or lack thereof) of the sponsor, client manager, project manager, and project team. Adding a project volatility risk construct. This construct, developed from a prior analysis (Sauer et al. 2007) includes changes in project personnel, project targets, and unexpected exogenous changes. Project volatility risk is an emergent risk. Testing this model using data from 194 IT projects, we found it was able to explain 39 percent of the variance in process performance, whereas the traditional model depicted in Figure 1, using the same data, could only explain 16 percent of variance (Gemino et al. 2006a, 2006b). The good news here is that the form of interactive modeling we have pioneered appears to generate strong explanatory power for process performance. The bad news is that it is the wrong dependent variable. As we argued earlier, what counts today is value achievement. To the extent that we have been able to model value achievement (i.e., product performance), we are only able to explain 20 percent of the variance. This has caused us to reflect on our approach to this research. And, to the extent that IFIP WG 8.6 should be Hkewise concerned with understanding project value, the questions and issues we have raised should be of interest. 500 Part 7: Position Papers We have some key questions we are asking ourselves and hope they will be of interest to the IFIP WG 8.6 audience. They deal with the issues of conceptualization and measurement. 1.1 Conceptualization of Business Value What do we mean by business value? And to whom? In the National Health Service example, adoption of the technology supporting electronic booking of hospital appointments might lead to new value for patients, health managers, and clinicians. It could generate new costs both for these stakeholders and for others such as primary care trusts who gain no direct benefit. Is value the difference between benefits and costs? Do we measure only in economic terms? Any technology-enabled change generates many layers of effect—and each effect can generate more effects. Deciding which effects to track and which to ignore is a difficult research problem. And should we include the symbolic value of new IT in motivating and giving staff a sense that they are members of a progressive organization? Or again, what about benefits in terms of future flexibility deriving from a new platform? Our research team is wrestling with these kinds of problems in trying to determine where to draw the boundaries for a theoretical conceptualization of business value. 1.2 Measurement of Business Value Assuming we could agree on a definition of business value and, therefore, where we might start to measure it, there are significant issues still to face. One approach to measurement of benefits is to make comparisons between expectations and actual achievement. That way we could say that 108 percent of the expected economic benefits and 23 percent of the expected health benefits were achieved. That approach leaves us with the problem of whether different outcomes should be equally or differentially weighted. It also leaves a question about the baseline expectations. Is it original expectations that matter or subsequent modifications? Our research (Gemino et al. 2006b) has found that targets (i.e., scope, time, budget) change on average eight times in an IT project, so determining a baseline for measurement is problematic. If the business conditions change, modified targets may be justified as a new baseline. If the project underperforms and targets are adjusted downward to reflect this situation, it is less obviously appropriate to use the modified targets. There is also the question of who is authoritative as a provider of information about value and at what stage to collect it. So, can a project manager provide us with reliable data about benefits or must we ask the stakeholders? Which stakeholders? When do we ask? A month after implementation, a year after, or longer? As the period lengthens, more extraneous factors affect the value experienced and stakeholders' perceptions. It is clear then that there are some serious questions to be answered if researchers are to model and measure the antecedents of business value. One possible reason why we have thus far been unable to explain much variance on value-related project outcomes is that traditional research on the antecedents of perfor- Sauer et al./The New Challenge of Business Value 501 mance have concentrated on risks and project management practices more relevant to budget and schedule targets. Traditional thinking on project management practice has focused on a restricted set of behaviors relating to control and coordination such as administrative control, team-building, and knowledge management (pace the value management literature in engineering management). We need a more fundamental reexamination of project performance and management practice. We must ask project managers what they do differently to achieve outcomes beyond the traditional targets. We may have to frame elements of project management in entirely new ways. For example, it has been commonplace that time, cost, and quality are often traded off against each other. In the future, it may prove preferable to think of them not as targets but, say, as repositories of slack resources that can be accessed to help manage changing requirements. Or, it may be that project managers who secure value are obliged to embrace changing requirements rather than resist them. The whole emphasis on process control may need to be re-thought. 2 SUMMARY Our argument is that research into project performance and adoption and diffusion face a shared problem. Both fields may need to move away from reliance on intermediate measures and toward researching delivery of business value and its antecedents. We have tried to show that there are some substantial issues that will require serious research effort to resolve. We want to push the argument further by suggesting that the project manager has not figured as a major unit of analysis in adoption and diffusion studies, but that given the increased requirement for project managers to deliver targets that go beyond implementation, it would be appropriate to rectify this. Equally important is the absence of the adoption construct in project performance literature. There may be an important opportunity for project management research to incorporate the findings relating to adoption and diffusion into our models. Our proposal is that investigators of both IT project performance and adoption and diffusion could usefully combine forces to develop more comprehensive models that are methodologically rigorous and more directly useful to practitioners. References Gemino, A., Reich, B. H., and Sauer, C. "Factors Influencing IT Project Performance," in Proceedings of the 12'^' Americas Conference on Information Systems, Acapulco, Mexico, August 4-6, 2006a, pp. 3733-3740. Gemino, A., Reich, B. H., and Sauer, C. "Inside the Black Box: Developing a Model for IT Project Performance," unpublished working paper, Simon Fraser University, 2006b. Reich, B. H. "Managing Knowledge and Learning in IT Projects: A Conceptual Framework and Guidelines for Practice," Project Management Journal, 2007 (forthcoming). Reich, B. H., and Wei, S. Y. "Searching for Knowledge in the PMBOK Guide," Project Management Journal (37:2), June 2006, pp. 11-27. Sauer, C, Gemino, A., and Reich, B. H. "Managing Projects for Success: The Impact of Size and Volatility on IT Project Performance," Communications of the ACM, 2007 (forthcoming). 502 Part 7: Position Papers About the Authors Chris Sauer is a fellow in Information Management at Oxford University's Said Business School, University of Oxford, Egrove Park, United Kingdom. He was Secretary and then Deputy-Chair of WG8.6 from 1994 through 2001. Chris can be reached by e-mail at Chris.Sauer@sbs.ox.ac.uk. Blaize Horner Reich is professor in management information systems of the Faculty of Business Administration at Simon Fraser University, Vancouver, Canada. Blaize can be reached by e-mail at breich@sfu.ca. Andrew Gemino is an associate professor in management information systems of the Faculty of Business Administration at Simon Fraser University, Vancouver, Canada. Andrew can be reached by e-mail at gemino@sfu.ca.