International Journal of Medical Informatics 97 (2017) 86–97

Contents lists available at ScienceDirect

International Journal of Medical Informatics

journal homepage: www.ijmijournal.com

Review article

A literature review for large-scale health information system project

planning, implementation and evaluation
Judith Sligo (PhD) (Research Fellow) a,∗ , Robin Gauld (Professor) a , Vaughan Roberts b ,
Luis Villa c
a
Department of Preventive and Social Medicine, Dunedin School of Medicine, University of Otago, Dunedin, New Zealand
b
Healthy Together 2020 Technology Programme, Counties Manukau Health, New Zealand
c
Research and Evaluation Office, Health Intelligence and Informatics, Ko Awatea, New Zealand

a r t i c l e i n f o a b s t r a c t

Article history: Information technology is perceived as a potential panacea for healthcare organisations to manage pres-
Received 14 July 2016 sure to improve services in the face of increased demand. However, the implementation and evaluation
Received in revised form 6 September 2016 of health information systems (HIS) is plagued with problems and implementation shortcomings and
Accepted 20 September 2016
failures are rife. HIS implementation is complex and relies on organisational, structural, technological,
and human factors to be successful. It also requires reflective, nuanced, multidimensional evaluation to
Keywords:
provide ongoing feedback to ensure success. This article provides a comprehensive review of the liter-
Health information systems
ature about evaluating and implementing HIS, detailing the challenges and recommendations for both
Implementation
Evaluation
evaluators and healthcare organisations. The factors that inhibit or promote successful HIS implemen-
tation are identified and effective evaluation strategies are described with the goal of informing teams
evaluating complex HIS.
© 2016 Elsevier Ireland Ltd. All rights reserved.

1. Introduction increase patients’ access to health services, remote care and conti-
nuity of services [2–6]. HIS would be expected to have social and
This literature review was instigated with the specific purpose economic benefits for patients, families and healthcare providers
of providing the background for an evaluation project being under- [7].
taken by the authors. During the process of reviewing the literature However, current rates of adoption of health information
the authors became aware that there was currently no comprehen- technology are low [8] and health information systems are under-
sive review of the crucial pre-requisite and likely inhibitory factors utilised [9]. Healthcare is slow to adopt technology compared to
of successful implementation of complex health information sys- other industries [10]. An examination of HIS adoption in seven
tems (HIS) that linked this information with their evaluation. This industrialized nations showed that while many nations have
review builds on other reviews in the literature with this summary achieved high levels of primary care electronic health record (EHR)
and linkage. adoption, they all lagged with respect to inpatient EHRs and health
Healthcare organisations with limited financial resources are information exchange systems [11]. Technology is frequently not
faced with mounting pressures in response to epidemiological and well accepted, used sparingly or not at all [12] and HIS has been
demographic changes. Governments, policy makers, information described as “high tech with a low impact”[13]. There are serious
technology businesses, healthcare organisations and consumers issues with the implementation of HIS and reports of HIS imple-
have expectations that these challenges can be addressed via mentation failure are not hard to find in the literature [9,14–21].
technological innovations. Health information systems (HIS) have Implementation is defined as the process of planning, testing,
potential to increase efficiency and save considerable amounts of adopting, and integrating HIS so that the technology becomes rou-
health expenditure [1]. Effective technology can reduce clinical tinely used in the organisation. Kaplan [22] notes that “there has
errors, support clinicians, improve information management and been a long history of difficulties in achieving clinical use of some
kinds of clinical informatics applications. ” Within this context, it
is imperative that HIS implementation is evaluated and features of
successful implementation identified.
∗ Corresponding author.
E-mail address: judith.sligo@otago.ac.nz (J. Sligo).

http://dx.doi.org/10.1016/j.ijmedinf.2016.09.007
1386-5056/© 2016 Elsevier Ireland Ltd. All rights reserved.
 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97 87

The authors of this article are currently immersed in the evalu- encapsulated all of this diversity in the literature, fourteen further
ation of an ongoing HIS intervention which is being implemented systematic reviews were identified [3,5,6,10,24–33].
in parallel with an organisational redesign process across a hos- In total 367 relevant publications were identified and read.
pital and associated health care services, including primary care. Using these two search strategies enabled the authors to focus
The process of the implementation of technology in this setting on factors that were particularly relevant for the evaluation of the
has evolved and changed course, requiring the evaluation to evolve implementation of this specific transformational change and HIS,
in parallel. Reviewing the international literature was paramount while also identifying and detailing factors that promote or inhibit
in order to provide background for understanding the complex, HIS implementation in general.
nonlinear, and unpredictable nature of HIS implementation. The
literature review, therefore, became a key part of the evaluation 3. Definitions
methodology. The literature review specifically focused on the
potential challenges and benefits of implementing HIS and the Information technology innovations in healthcare organisations
difficulties in evaluating such implementations. In this regard, it encompass an enormous range of technologies, which is rapidly
represents the most up-to-date review for those evaluating large- increasing in number and diversifying in purpose [3,34,35]. Hersh
scale HIS projects. This article summarises that literature in order to [27] noted that there were 450 telemedicine programmes avail-
assist other researchers embarking on similar evaluation endeav- able worldwide at the time of writing. Rye and Kimberley’s [31]
ours. systematic review noted that there were 255 unique innovations
This literature review has five sections: the methods provide described in 55 studies. There is significant variation in terminol-
a brief overview of the literature search strategy, which is fol- ogy around this technology: the all-encompassing term ‘eHealth’
lowed by a brief explanation of definitions used in the literature. is used [36–40], which may refer to a range of technologies and
The next section backgrounds the meaning, purpose and types of systems [13,41].
ehealth interventions, and summarises the literature evaluating Others use the term ‘telemedicine’ [9,27,42] which, in sim-
the effectiveness of health information technology initiatives. The ple terms, refers to “the delivery of medical health services at a
third section addresses the need for evaluation of health informa- distance, [although] there is no single or uniform telemedicine
tion technology and its implementation, and outlines some of the application” [43] and includes a range of technologies. ‘Telehealth’
challenges involved in evaluation and recommendations for qual- [7,44–46] is another term introduced to reflect a broader scope of
ity evaluation. Next, the factors which have been shown to increase health-related functions such as education and administration [47].
the likelihood of successful implementation of health information The literature also includes terms like ‘health information tech-
technology innovations are summarised before the final section nology’ (HIT) [2,24,26,32,48–55] or ‘health information systems’
describing the factors that inhibit successful implementation. (HIS) [2,17,51,54,56–61] and variations on these terms like health-
care information networks [62], healthcare innovations [63,64],
and healthcare transformations [65,66]. They may be linked to the
2. Method setting, as in ‘hospital information systems’ or ‘hospital information
technology’ [60,67–70]. Thus, the terminology in this field of study
This literature search had two strategies undertaken by the suffers from a lack of clarity and an absence of agreement about
first author in consultation with the other authors. The first was a the definitions of the concepts [47], which has led to uncertainty
systematic search of nine general and medical databases for trans- among academics, policymakers, providers and consumers.
formational/complex and program/me, hospital/healthcare and In the absence of any consensus around a consistent use of ter-
IT/IS (or technology). The focus was on large scale and transforma- minology, this literature review uses the term ‘health information
tional technology implementation with the goal of identifying the systems’ (HIS) to encompass the widest range of possible infor-
multiple factors involved in successful implementation of complex mation technology used in healthcare systems. Yusof et al. [71]
HIS. The intention was to avoid the plethora of literature regarding provides an overview of the meaning of HIS:
implementation of technology devices and small systems where
implementation is more straightforward and complexity is less of HIS assist healthcare organisations to gather, process, and
a concern. The search strategy is detailed in Appendix A. Across each disseminate information within the organisation and their
database, several thousand results were retrieved. Abstracts were environment. HIS incorporates a range of different types of
then reviewed and details of pertinent articles included in an end- systems, which include patient information systems, adminis-
note database. In this filtering process special attention was paid to trative systems, radiology and pharmacy information systems,
successful implementation of complex, large healthcare technology telemedicine and hospital information systems, such as com-
initiatives in order to identify positive models for the intervention puterised physician entry systems.
team. Evaluation literature was also reviewed and detailed. In our
search for positive examples of the implementation of complex, 4. Rationale and effectiveness of health information
transformation HIS we encountered an abundance of examples systems
of unsuccessful projects and challenges to implementation, which
allowed us to also identify the factors that are likely to impede suc- Researchers, health professionals, patients and policy makers
cessful HIS implementation along with the factors that promote have high expectations of information technology in health care
successful implementation. organisations and there is considerable international interest in
A second strategy was similar to that used by Potts et al. [23], exploiting the potential of HIS [6,16,25]. The World Health Organi-
which they call a ‘meta-narrative literature review’. This involves sation has an eHealth unit and the 58th World Health Assembly in
an exploratory informal searching phase followed by identifica- Geneva in 2005 recognised the potential of eHealth to strengthen
tion of key concepts, theories and preferred methods to make health systems and to improve quality, safety and access to care,
sense of complex, heterogeneous bodies of literature. The ini- and encouraged Member States to take action to incorporate
tial search described in the paragraph above indicated that the eHealth in health systems and services [72].
literature included theoretical and evaluative articles which had Eysenbach [73] summarises the goals of technology in health-
methodological and design differences from different disciplines care, suggesting that eHealth should be: (1) Efficient, thereby
and theoretical perspectives. To ensure that this literature review decreasing costs, (2) Enhance quality of care, (3) Evidence based,
88 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97

proven by rigorous scientific evaluation, (4) Empowering for con- provided by technological innovations is any better than tradi-
sumers and patients, (5) Encouraging a partnership relationship tional methods [23,25–28]. Whitten’s [33] systematic review of HIS
between patient and health professional, (6) Educate physicians cost-effectiveness found that there is no conclusive evidence that
and consumers, (7) Enabling information exchange and communi- telemedicine is a cost effective way of delivering healthcare. Mis-
cation in a standardized way between health care establishments, try [29] reviewed the cost-effectiveness literature ten years later
(8) Extending the scope of health care beyond its geographical and and concluded that the results of their review were consistent with
conceptual boundaries, (9) Ethical – e-health involves new forms of previous findings: there is no further conclusive evidence that tech-
patient-physician interaction and poses new challenges and threats nological interventions are cost effective compared to conventional
to ethical issues, and (10) Equitable. Haux [59] identified seven gen- healthcare. However, it is also the case that these reviews noted
eral tasks of HIS over time. These are [59]: (1) to move paper-based methodological shortcomings in studies evaluating cost effective-
processing and storage to computer-based; (2) to move from local ness. These were particularly around the amount of methodological
to national and global HIS; (3) to include patients as HIS users; detail provided and the methods used to measure cost effectiveness
(4) to use HIS data for healthcare planning, clinical and epidemio- [29,33].
logical research (aside from patient care and administration); (5) While there is also some evidence that complex HIS can be suc-
to change the focus from technical aspects of HIS to management cessful [39,83–87], Heeks [17] attributes the number of accounts of
change and strategic information management; (6) to place more successful HIS to the negative bias against publication of failures,
emphasis on image and molecular data; and, (7) to acknowledge and it is noted that most accounts of successful health information
the steady increase of new types of technologies, perhaps as yet technology are from specific applications in a single location [8,10]
unimagined. creating serious doubts about the generalisability of the HIS bene-
fits recorded in these (often elite) settings [32]. The scientific basis
4.1. The rationale for HIS of claims of improved care have not been consistently established
and evaluation studies challenge existing findings as often as they
The reasons cited in favour of the implementation of HIS are corroborate them [88]. McLean [28] concludes that “Policy makers
primarily around efficiency, cost, quality and safety [4,26]. Health- and planners need to be aware that investment in telehealthcare
care organisations are increasingly required to address the growing will not inevitably yield clinical or economic benefits.”
costs of delivering healthcare to aging populations without com- However, systematic reviews indicate that there does appear
promising quality, access, or equity, and are required to integrate to be evidence of some increase in quality improvement as a
new scientific evidence into practice [25,39,74,75]. There is some result of HIS. Chaudhry et al. [25] reviewed 257 studies from
evidence to suggest that HIS can lower costs, increase efficiency 1995 until 2004 and found that there were some benefits to
and productivity and provide a positive return on investments quality, primarily in the domain of preventive health along with
[1,3–5]. Implementing HIS presents potential for increased safety increased adherence to guideline-based care, enhanced surveil-
[76] and improved clinical practice via reduced clinical errors, lance and monitoring, and decreased medication errors. The major
fewer adverse drug events, increased availability of current qual- efficiency benefit shown was decreased utilization of care but the
ity patient information, and clinician diagnosis support Koppel [2]. lack of data around cost effectiveness was noted. However, 25%
Benefits for patients include having greater engagement with their of the articles reviewed came from only four academic institutions,
own care [77], monitoring of chronic illness, disease prevention termed ‘health IT leaders’ by Chaudhry et al. [25]. These institutions
and increased efficiency in hospital settings [1]. HIS makes home- had implemented internally developed systems and these authors
based care for patients in remote areas or with chronic conditions queried whether other institutions would be able to achieve simi-
a possibility [49,78] and social media has been heralded as a means lar benefits. Goldzweig et al. [26] replicated Chaudhry et al.’s [25]
of transforming healthcare practices [79]. HIS implementation can methodology and reviewed 179 studies published between 2004
lead to positive changes in the organisation and delivery of health- and 2007. They found a proliferation of ‘stand alone’ applications
care [23,39,80]. that had mixed evidence of benefits. Like Chaudhry et al. [25],
Goldzweig et al. [26] noted the paucity of cost-benefit data, and
4.2. How effective are HIS? in reviewing the refinement of interventions in the health IT leader
institutions noted that most of the new studies reported few ben-
A review of the literature on the effectiveness of HIS requires efits from the HIS.
a reflection on the concepts of ‘failure’ and ‘success’. Doherty et al. Buntin et al. [24] used the same inclusion criteria as Chaudhry
[80] suggest that one of the reasons that information systems across et al. [25] and Goldzweig et al. [26] to review 154 articles from 2007
different settings are seen as ‘failing’ is because of the way success until 2010 and categorised the articles as positive (improvement in
and failure are perceived by different stakeholders. They note that one or more aspects of care and no worse), neutral (no demon-
IT project management will judge an intervention as successful if it strated change in care or care setting), mixed positive (an overall
is delivered on time, on budget and to specification while the organ- positive conclusion: generally about 3 positives for each negative)
isation will perceive success in the ultimate delivery of the intended and negative. Sixty two percent were evaluated as positive and 92%
benefits (for example, usability, efficiency, safety). The issue of the considered positive or mixed-positive. Unlike the earlier reviews,
definitions and implications of how these terms are understood the results for the projects describing ‘health IT leaders’ projects
is revisited below in the section “Characteristics of Successful HIS no longer differed from other studies. These authors conceded that
Implementation”. there is likely to be positive bias due to non-publication of negative
No matter how success or failure is defined (if it is) the evidence results. It is also the case that their methodology for assessing the
of effectiveness is generally weak and inconsistent [6]. Informa- literature varied from the earlier systematic reviews and appears a
tion systems of all types notoriously fall short of their expectations rather crude measure of success. However, the evidence from the
and fail to deliver benefits (see for example, Gauld and Goldfinch Buntin et al. [24] review indicates that not only health IT leaders
[81]. Shpilberg et al. [82] reported that only 15% of business exec- are experiencing the positive effects of HIS. Thus, there appears to
utives surveyed believed that their company’s IT capability was be some scope for cautious optimism around the potential benefits
highly effective, ran reliably, and delivered projects with promised of HIS. This does need to be viewed in the light of other reviews
functionality, timing, and cost. Systematic reviews of healthcare that find little evidence that HIS are cost effective or provide bet-
settings consistently find that there is little evidence that care ter quality healthcare. However, it is also important to remember
 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97 89

that absence of evidence does not equate with evidence of ineffec- them to support improved healthcare [108]. This was prompted
tiveness [6]. Perhaps the best summary of the evidence regarding by the concern that merely adopting EHRs was inadequate to sub-
the effectiveness of HIS comes from Karsh et al. [8] who state that stantially improve care [109]). The majority of HIS have not been
“current research demonstrates that health information technol- developed using the rigorous software engineering methods used
ogy (HIT) can improve patient safety and healthcare quality in in other safety–critical environments such as the airline or railway
certain circumstances.” signalling industries [12]. Thus, implementing HIS is a risky busi-
There is ample evidence that unanticipated consequences ness. This perhaps explains, at least in part, why some organisations
of implementation of HIS can occur [89–95]. Blosomrosen [96] might be tentative venturing into an expensive HIS implementa-
categorises unintended consequences four ways: desirable or tion.
undesirable, anticipated or unanticipated, direct or indirect, and To summarise, current research suggests that HIS can be
latent or obvious. Unanticipated and undesirable consequences can effective at improving healthcare and patient safety in certain cir-
undermine patient safety practices and occasionally harm patients. cumstances, although there is no convincing evidence that they
Different types of technological interventions and devices can have are cost-effective compared to conventional modes of delivering
different types of unanticipated results and may result from user healthcare. HIS are complex and may have unforeseen outcomes,
error or workarounds [8,97]. For example, Ash et al. [98] describe which can be positive or negative. There are high risks in imple-
the ‘silent errors’ in the use of patient care information systems menting HIS and implementation levels remain generally low [11].
(PCIS), which occurred in the process of entering and retrieving
information, and in the communication and coordination pro-
5. Evaluation
cess that the PCIS is supposed to support. Computerized provider
order entry (CPOE) implementation can have clinical unintended
Effective evaluation allows us to understand how and under
adverse consequences such as more or new work for clinicians,
what conditions HIS work, and determine the safety and effec-
unfavourable workflow issues, continual system demands, prob-
tiveness of the system [2,71,110]. Evaluation can provide guidance
lems with paper persistence; negative changes in communication
to the implementation process and mitigate unplanned negative
patterns and practices, user dissatisfaction, errors, unexpected
outcomes [17,111]. Ammenwerth [112] defines evaluation as “the
changes in power structures and overdependence on the technol-
act of measuring or exploring attributes of a health information
ogy [99]. The compromise of patient confidentiality and privacy is
system (in planning, development, implementation, or operation),
another unintended consequence that can occur with the imple-
the result of which informs a decision to be made concerning that
mentation of new technology, particularly record keeping [100].
system in a specific context.” This should include the inevitable
Unanticipated consequences can also produce desirable results,
organisational change which accompanies the implementation of
such as improved communication or new and enhanced work sys-
HIS [80]. Early approaches to evaluation focused on the “measure-
tems [101,102].
ment of changes in processes and of the consequences of these
changes” [113] while more recently attention has been paid to
4.3. Implementing HIS
the complex, iterative and multidimensional implementation pro-
cess [30]. Effective evaluation accompanies the whole life cycle of
Implementation of HIS is not a simple straightforward linear
HIS [2,17], evaluating technology against a comprehensive set of
process. Adoption and implementation of HIS are not the same
measures throughout all stages [6].
thing [31]: just because an HIS has been adopted, it does not nec-
Measuring the success of HIS is not straightforward and the
essarily mean it is being used (or used in the way it was intended).
challenges in the organisation and setting of HIS make both imple-
Successful implementation may involve a lengthy process starting
mentation and evaluation of the HIS difficult. Evaluation processes
with planning, designing, and piloting before moving into being
are also often flawed. The recognised difficulties with evaluations
(possibly intermittently) used, modified, accepted (or not) until it is
are briefly outlined below.
considered routine [10,103]. The more comprehensive the technol-
ogy, or the wider the span of the implementation, the more difficult
it appears to be to achieve success [20]. A healthcare organisation 5.1. Healthcare settings and the nature of HIS provide challenges
encompasses a complex web of inter-related clinical, cultural, and for evaluation
technical issues situated within a wider societal and political envi-
ronment. Implementing HIS is not merely about technical issues, Healthcare settings are complex. A complex system is one that
it is about complex and multidimensional organisational changes adapts to changes in its local environment, comprises other com-
[25]. Car et al. state that “even when high quality interventions are plex systems [such as people] and acts in a non-linear fashion
developed, they frequently fail to live up to their potential when [114,115]. Healthcare settings are multiprofessional organisations,
deployed in the ‘real world’ [36]. which often include a dual hierarchical structure involving clini-
Implementing HIS is expensive [1]. Most countries dedicate cians and managers. Clinicians also have a high degree of autonomy
not less than 2 to 6% of their health budget to information tech- (collectively and individually) and decentralised decision making is
nology [2,4]. In one survey, the majority of both economists and common [30]. Healthcare organisations operate at different levels
physicians identified technological change as the primary reason and may involve several linked institutions, a large conglomeration
for the increase in the health sector’s share of GDP in the last 30 of small healthcare settings, single hospitals with many depart-
years (Fuchs, cited in Rye and Kimberley) [31]. There are other ments, primary care organisations and/or any combination of these.
costs involved in the implementation of HIS: it can be hugely dis- These may be funded and administered in different ways and have
ruptive for staff and organisations [77,104–106], since it can be a links to other organisations at the same and different levels. Health-
profound agent for change [12,18]. Despite evidence that HIS can care organisations are also responsible to their funders, which
impact negatively or positively on patient care and the identifica- may include private, industry, or governmental bodies. Healthcare
tion of health information technology as being amongst the least organisations are likely to experience competing tensions between
reliable [107] there are no regulatory requirements to evaluate HIS their obligations at these different levels when implementing HIS
safety [8] (although in the United States the Health Information [23]. Such complexity can cause complications for HIS implementa-
Technology for Economic and Clinical Health Act, 2010 identified tion, including multi-stakeholder perspectives, power asymmetry
a set of objectives for the ‘meaningful use’ of EHRs, which enable and politically led changes [34].
90 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97

HIS are also complex [115]. HIS implementation needs to be [10,13,30,31,119]. Robert et al. [30] identify two approaches to eval-
understood and evaluated as a socio-technical process [71]. Imple- uation, one which views the implementation of HIS as a sequential
mentation of HIS usually involves the procurement of technology process of stages of implementation where organisational vari-
and complex support systems from outside organisations, which ables are associated with higher or lower rates of adoption, and;
add extra complexity to managerial and governance structures. The a more complex, iterative and multidimensional process. This sec-
healthcare organisation’s existing IT infrastructure is required to ond approach to evaluation allows the evaluation to incorporate
be responsive and become compatible with the new HIS [17]. HIS the social, political, organisational and related processes as they
may comprise different types of devices, technology and be used for unfold over time [22]. There is an increasing acknowledgement of
any number of healthcare functions from imaging to communica- the importance of context in the evaluation literature [34].
tion to prescribing to recording to diagnosis. Technology is rapidly Theoretical perspectives are increasingly being used to provide
changing and healthcare organisations are not equipped for quick a framework for understanding HIS implementation. This includes
changes with their highly institutionalised structures and practices actor network theory (ANT), diffusion of innovation theory, the
[16]. All these features of HIS make their implementation into an theory of fit between individuals, task & technology (FITT), and
already complex healthcare system challenging. social construction of technology theory [103,104,117,120–124].
Challenges for evaluations include attributing causality in a sys- Frameworks for understanding the interdependencies between the
tem which is dynamic and non-linear. Consequences are likely to technological, human, and socioeconomic environments provide
be multiple and have spin off effects requiring measurement of out- structures for understanding the factors that promote and inhibit
comes at multiple levels [115]. This also means that evaluations are implementation of HIS and the conditions under which HIS provide
likely to be time-intensive and expensive. Outcomes are difficult to benefits (see van Gemert-Pijnen et al. for a systematic review of this
quantify [34]. Evaluating HIS implementation combines healthcare, literature) [13].
information technology, biomedical science and medical informat-
ics [104] and may require knowledge of human factors/systems
5.3. Recommendations to improve the quality of HIS evaluations
ergonomics, organisational/occupational/social psychology, man-
agement (particularly organisational change management) and
Because of the problems described above with the consistency,
information systems [10]. Perhaps because of these difficulties, the
methodologies, and lack of depth and breadth of HIS evaluations,
quality of studies evaluating HIS implementation is generally low
recommendations to address these difficulties have been identi-
[6,17,27,28,31,33].
fied. Evaluations should systematically assess the impact of the HIS
on various stakeholders over time [2,75,110], clarifying goals and
5.2. Evaluation of HIS has varied in quality
theoretical perspectives [13]. The process of implementation is not
static but is vulnerable to changes in the organisation and larger
The body of evidence about the success of HIS and its implemen-
context, requiring the evaluation to adapt and change in synchro-
tation is plagued by variation in the quality, methods, theoretical
nisation with the implementation, paying particular attention to
approaches and multidisciplinary perspectives of evaluation stud-
the sequence of the implementation [113,125–127] and the dif-
ies. While variation in approaches and perspectives can be useful
ference between technology design and reality of implementation
given the range of HIS, implementation stages and processes, eval-
[17].
uation disparity makes it difficult to draw conclusions about what
The gap between the individual(s), task and technology should
effective HIS implementation looks like. Even the conceptualisa-
also be addressed in evaluations [13,39,71,128,129]. Other sugges-
tion of implementation of HIS is not consistent: it is variously
tions for improving evaluation include meticulous documentation
perceived as adoption, deployment, diffusion, implementation,
of assessment methods and research decisions made [2] and paying
infusion, integration, normalisation, embedding or routinisation
more attention to stakeholders’ disengagement from HIS [31]. To
[10,64,75,116,117] depending on the perspective and timing of the
summarise this literature: well-funded, meticulously planned, rig-
evaluation. Brender [110] notes that evaluation is required at all
orous, multidisciplinary research which assesses outcomes across
phases of the development and the purpose of evaluation depends
the multiple dimensions of HIS impact (human, technological, con-
on whether the HIS is in its explorative, technical development
textual) throughout the life cycle of the HIS (from conception,
or adaptation phase. The heterogeneity of terminology, method-
through design and implementation, to maintenance) is required
ology and design means that it is difficult to draw conclusions
[2,31,115,127,129]. Although there are shortcomings with the eval-
about the effectiveness of HIS or the factors that promote or inhibit
uations of HIS implementation, the evaluative literature is large and
their implementation [7,23,31,33]. Systematic reviews note the
has been able to identify factors that make successful implementa-
anecdotal nature and retrospective nature of HIS evaluation, often
tion more likely. These are discussed below.
based on case studies or implementation of small interventions in
single organisations [10,25,27] or on expensive, tailored hospital
based HIS [25,26,31]. Evaluation studies are often short term while 6. Characteristics of successful HIS implementation
HIS implementation is a lengthy process [25,33,115]. Few studies
include a cost benefit analysis [6,25,26,28]. When cost-benefit anal- Evaluating the implementation of HIS as successful (or not) is
ysis is included, it is generally restricted to simple cost comparisons not straightforward. Evaluation is subject to the vagaries of tim-
from a narrow perspective [7,29,33,118], focusing on whether the ing: today’s HIS success may be tomorrow’s HIS failure and vice
project came in within budget [80] or describing potential benefits versa [17]. The use or non-use of the technology is not a criterion
from economies of scale [33]. for success [8]. For example, technology can be used incorrectly or
Much of the criticism of HIS evaluations is that they do not create more work for the user indicating that use is not necessar-
address the complexity of the context of an HIS implementation ily an endorsement that the HIS is successful. The complexity of
and that measurement is simplistic [5,6,25,31,119]. Kaplan [22] HIS implementation means that successful implementation can be
notes that studies with pre-specified dependent measures miss interpreted differently by various stakeholders [75,80].
unanticipated consequences and enduring emergent effects. Sys- While it is true that many evaluations do not explicitly define
tematic reviews claim that evaluations of HIS implementation are ‘success’, quantitative evaluation literature generally measure out-
under-theorised and have not adequately addressed the relation- comes of HIS compared to its original goals [6] and qualitative and
ship between the technological, human and contextual features mixed methods evaluations are more likely to measure ‘success’
 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97 91

contextually [13,23]. It is also true that the correlates of successful bination of factors in which the government played an important
HIS inevitably fall into human, technological and organisational or but indirect role: requirement of GPs to submit claims and capture
contextual domains [3,10,31,71]. These factors are interrelated and other data electronically”. Implementation of HIS can also involve
have complex relationships with each other and there is an increas- legal, administrative, and governance issues that require higher
ing understanding that the degree of alignment between them is of level communication and support [100].
prime importance [10]. The factors associated with successful HIS Cresswell and Sheik [10] remind us that organisations and their
implementation are discussed below within the classifications of structures are made up and determined by people and that these
these three dimensions. It is important to bear in mind that these dimensions are closely related. The human factors that are likely to
categories are not discrete. facilitate successful HIS implementation are addressed below.

6.1. Structural/contextual/organisational factors that promote

successful HIS implementation

This category includes everything that needs to be in place prior

to and during the implementation of HIS. Factors in the organisa-
tional dimension are crucial6 but have a tendency to go unnoticed 6.2. Human factors that promote successful HIS implementation
because precedence is given to technical and clinical characteris-
tics [74]. Robert et al. [30] identify eleven key characteristics in an Human factors include those characteristics and issues that
organisation to be ready for assimilating innovations. These are: (1) make individual users more likely to accept and operate technology
Money, staff and other resources are available; (2) A clear division of as well as concerns with human resource factors such as person-
labour between departments and units, with each concentrating on nel, training and relationships between individuals. Li et al.’s [5]
its own strengths; (3) Decentralised management; (4) People in the systematic review of the eHealth adoption literature summarised
organisation are familiar with basic concepts and can apply them to characteristics of healthcare providers that make them more likely
a new project; (5) Staff have the skills and capacity to horizon-scan to accept and use technology. They acknowledge that there were
and capture new ideas; (6) Knowledge sharing within and out- some contradictions in the results but identified some characteris-
side the organisation is enabled and promoted by senior staff; (7) tics that were noted across much of the literature. These included
Top management provides strong and competent leadership and having some previous experience with technology, having a per-
vision; (8) Good relationships and communication at middle man- ception that using the technology was not voluntary or optional
agement level; (9) A risk-taking climate where people are rewarded and perceiving that the technology would be easy to understand
for taking risks; (10) Goals and priorities are clearly articulated; and use. It was also helpful if users judged themselves as capa-
and, (11) There is timely feedback on the success of innovations. ble of operating the new system. People were more likely to use
Visionary leadership of the organisation and implementation pro- the system if they deemed it to be effective and better than previ-
cess are also key factors in the success of HIS implementation ous systems or workflows. It was also essential that new users did
[130–134]. Perhaps the provision of adequate funding to purchase not perceive the innovation as time consuming and that it should
and implement the HIS is the primary organisational requirement enhance their work. If individuals thought that their colleagues or
[5,39,100]. significant others should adopt the system, they were more likely
Communication between levels of the organisation and between to use it. Users also required adequate training and time to become
management, clinicians and information staff is crucial to the suc- familiar with the new system. Time and effort involved in learning
cess of HIS implementation [9,77,135–138]. This includes ensuring new technologies can be a significant barrier to uptake [100].
that there is someone to liaise between IT staff, clinicians and The most commonly cited facilitating factor in the human
management, facilitating workflows and highlighting problems domain is the perception of the benefits of the innovation
[10,23,138]. Key organisational structures required for successful [3,5,10,31,75,144,145]. It is crucial to have the ongoing involve-
implementation of HIS include clear management and governance ment of key stakeholders, right from the conception and design
structures, task orientated structures (as opposed to output ori- stages [21,30,139]. The opportunity for end users to test proto-
entated structures), minimal staff turnover, good capacity of staff, types in the healthcare setting can help ensure technology is used
who are released from other duties and reimbursed for their role by professionals and patients [10].
in the implementation, realistic timelines, well arranged logistical Good communication is required so that information flows
procedures related to the innovation and an understanding that the from management and information technology staff to clinical and
implementation is an ongoing process [75,80,135,139,140]. administrative staff and that end-users are able to take ownership
Along with systems and structures within the organisation, of the process and new technology [87,140,145]. Dedicated effort
successful HIS implementation requires support from external to foster high levels of trust and early engagement with end users
organisations, which are also part of the organisational context. can bring about widespread ‘buy in’ from clinicians, patients, and
Central and local government support is integral: policy or per- other stakeholders [146]. Research evidence supports the benefits
sonnel changes (sometimes via elections) at these levels can of having ‘champions’, senior leaders who support the new system
completely destabilise HIS implementation or even be the cause providing information, support and take on the liaison role between
of its demise while strong support, communication and fund- management, technology staff and clinicians [3,10,75,77,147]. Paré
ing can provide strong impetus for successful implementation et al.’s [148] study of risk factors in implementing HIS found that
[16,139,141,142]. In New Zealand, where we are conducting an HIS project managers considered the lack of a project champion
evaluation that required this literature review, there is a centralised was the biggest risk to successful implementation.
government and public health system, but there are complicated Sufficient time for training and testing [117,138], which needs
and frequently changing health system structures [143]. An exam- to be well-resourced [84] is also important for successfully
ple of how complex historical and policy factors combine to implementing HIS. Information technology support for end-users
increase adoption of HIS (in this case electronic health records in assists confidence with the new technology [30,49,87,138]. The
ambulatory care settings) comes from Jha et al.’s [11] comparison fit between clinical work tasks and the design of the technology
of HIS adoption across seven nations. They found relatively high (workflow issues) significantly impacts on the likelihood of HIS
levels of EHR use in New Zealand, which they attributed to “a com- acceptance [145].
92 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97

6.3. Technical factors that promote successful HIS system. The literature identifies some factors that inhibit success-
implementation ful HIS implementation and these are briefly discussed below prior
to the conclusion.
Technical factors are intrinsically linked with the human factors
described above because people use the technology. The technical
domain is also immersed in the organisational structures described 7. Factors that inhibit successful HIS implementation
earlier. Bearing these connections in mind, it is possible to identify
features of the technological domain that can promote successful Some of the features of a healthcare organisation that are likely
HIS implementation. to hinder or constrain successful HIS implementation are contrary
Firstly, the existing ICT infrastructure must be able to assimi- and opposite characteristics to those described above as factors
late the new system. This requires the collection of baseline data promoting success. These ‘opposites’ include inadequate funding,
to determine the capabilities of the current infrastructure and lack of IT infrastructure, poor leadership, inadequate end-user
ensure that it is compatible with how it is expected to perform engagement and unrealistic timelines. The lack of compatibility of
with the new technology [17,21,128]. Technology implementa- the HIS with current work processes and the organisational culture
tion also needs to be managed effectively. Caccia-Bava et al. [149] and vision are commonly cited as factors that impede successful HIS
recommend that managers develop knowledge of how to get the implementation [3,30,57,62,150,163]. Here, attention is paid to the
best technology available, effective use of specific technologies, factors that are not related to those discussed above. These include
and benchmarking the use of specific technologies against lead particular challenges with technology, the factors that contribute
organisations. Information technology implementation teams are to user resistance, features of the organisation that make it difficult
required to provide sufficient on-site training as well as on-going to change and the importance of ongoing evaluation.
support as the implementation occurs [3,75,77,80,135,150]. The challenge to successful implementation is primarily around
A good ‘fit’ between the needs of the users and the technology the socio-technical and contextual domains. However, there are
is required [13,39,128] across the chronology of the implemen- risks to implementation that can be attributed to the technol-
tation – at pre, during, and post HIS implementation [151]. ogy. These include procuring a poor product, being lured into
This involves adequate design, testing, prototypes and the abil- the leading edge of technology that has not been adequately
ity to adapt the technology as required as this process occurs. tested, and insufficient long term IT planning, leading to piece-
The literature is consistent in finding that end-user involvement meal interventions that do not align with the organisation’s goals
across all stages is conducive to more effective implementation [21,162,164–166]. Innovation characteristics (safety, performance,
[26,35,54,134,140,152,153]. Although meeting the needs of the value, risk, characteristics, purpose) are intuitively linked to the
organisation appears to be one of the key factors in the techno- success of implementation, even if the evidence is inconclusive
logical domain, this needs to be balanced against the requirement [31]. This lack of evidence is partly because of the breadth and range
for interoperability. The need for interoperability should be taken of technologies, making it difficult to generalise.
seriously: interoperability of HIS in different organisations bridges Addressing the reasons people do not use HIS is also impor-
information gaps, reducing redundant clinical procedures and tant. “Resistance to change is a phenomenon that is so pervasive
increasing patient safety [154], accessing the full benefits of EHRs and widely recognised that it scarcely requires documentation”
[155] and potentially reducing costs [156,157]. However, there are [148]. Clinician resistance is a frequently cited risk to the suc-
challenges and barriers to effective interoperability (see Eden et al. cessful implementation of HIS [26,39,125,128,137], although this
[158] for a systematic review of these) and the quest for interop- is not necessarily merely a resistance to change per se. Morrison’s
erability can cause compromises to be made that are not beneficial [167] study of the implementation of a care records system in
to a local HIS [21,23,74,100,159]. the NHS found that staff were not reluctant users of the system
There are characteristics of the technology, devices and tools but the systems had poor functionality so were difficult to imple-
which make up the HIS that are important for successful imple- ment. Other studies have found that other end users resist new
mentation. They should be easy to use, clear and understandable; technology because it is time-consuming to use, cumbersome or
easy to learn to operate; flexible; and have easy navigation with because they have not been made aware of the benefits [168–170].
easy to remember tasks [160]. The technology should be intuitive, Clinicians may express concern about patient privacy and confi-
easily customised, have quality interface design and require little dentiality [28,144], a concern echoed by others in the healthcare
training [3,26,144,153]. organisation with a legal responsibility to ensure patient confi-
Communication is also crucial: between the design team and dentiality [101]. Clinicians also fear that new kinds of errors are
the implementation team [104] as well as the organisation and made because of clinical systems [98] and query the impact on rela-
the vendor [161]. The healthcare provider’s perception of the IT tionships with patients [137]. Technologies which inadvertently
provider as trustworthy and reputable is a factor in the suc- undermine perceived authority or professional autonomy are likely
cessful adoption of HIS [5]. Effective communication between to be resisted by users [10]. Suggestions for combating clinician
the information technology staff and the clinical and managerial resistance include initiating change with the technology which will
staff to ensure adequate staff training and support is also crucial provide the greatest benefit to clinicians [84] and providing finan-
[3,39,62,75,77,101,135,144]. cial incentives for training [5].
Health information technology is expensive, and requires ade- Successful implementation relies on the understanding of the
quate funding for successful implementation. Gabriel [49] suggests unique organisational structures and practices of healthcare organ-
that collaborating with other healthcare organisations can decrease isations, which are not characterised by an ability to integrate quick
costs of the technology via economy of scale. ‘Scope creep’ can also changes [16,21]. If the healthcare organisation’s leaders are risk
raise the risk of exceeding the budget [2,10,162] when the HIS is averse, there can be reluctance to adequately invest in the imple-
not thoroughly defined before the project’s start causing costs and mentation [21,128]. High staff turnover [75] and turbulence in the
timeframes to explode [148]. organisational environment [148] also create risks for the success-
It is clear that successful HIS implementation is a process that ful implementation of HIS. Likewise, when there are conflicting
involves organisational change. It requires more than software goals at different levels of the organisation, or if the information
delivery or the adoption of technology. This requires a healthcare technology and managerial teams are not aligned, it is unlikely that
organisational context and structures that are receptive to a new end-users will successfully implement a new system [84,92,163].
 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97 93

Morrison [167] also found that it was important for the organisa-
tion’s long and short term goals to be aligned. Summary points
The final factor that is likely to inhibit the successful imple- This article reviews and summarises the literature on the
mentation of HIS is not organising independent, robust, ongoing implementation and evaluation of health information systems
(HIS). The authors outline the challenges in implementation,
evaluation of the implementation [12,21,111], which is fed back
which have resulted in the failure of many HIS to be effectively
to the organisation and implementation team so they can respond
adopted and used in healthcare settings. Factors which pro-
as required and build a culture of learning. Reflexive monitoring mote and inhibit successful HIS implementation are identified
can reveal how users perceive and interact with the technology, from across the literature and are discussed within organisa-
provide appraisal of the costs and benefits, and provide feedback tional, technological and human domains. The HIS evaluation
to continually adapt the innovation and/or implementation on the literature is also summarised and recommendations for effec-
basis of evidence [124]. tive evaluation are detailed. This article provides a robust,
current review of the literature regarding all aspects of HIS
implementation and evaluation. This creates a foundational
8. Conclusion resource for organisations and evaluators implementing and
evaluating HIS.
Healthcare organisations are complex and under some pressure
to integrate technology into their practice in order to transform
care and become more efficient. However, evidence of the effec- transformation in the culture of the organisation, which takes time,
tiveness of technology in healthcare settings is mixed and the adequate resourcing, support and commitment at all levels.
uptake is generally low. Evaluating the implementation of HIS has
been historically inadequate, plagued by simplistic and diverse Author contributions
approaches making it difficult to generalise the results. Publication
bias has possibly created an unrealistic impression of the success Study conception and design: Sligo, Roberts, Gauld, Villa.
rates of HIS implementation. It is now recognised that a more Acquisition of data: Sligo.
nuanced approach to evaluation is required due to the organisa- Analysis and interpretation of data: Sligo.
tional change that is implicit in implementing HIS. Implementation Drafting of manuscript: Sligo.
of HIS is a socio-technical process and evaluations need to incor- Critical revision: Sligo, Roberts, Gauld, Villa.
porate organisational/contextual, human/social, and technological
dimensions. These dimensions are interrelated and co-exist but the Acknowledgments
exact nature of the relationship between the dimensions is less
clear and requires more attention [171]. Material in this article was generated as part of a large-scale
There is a large body of evidence about the implementation of health information system evaluation commissioned by Counties
HIS and some factors have been consistently identified as con- Manukau Health, Auckland, New Zealand.
tributing to successful implementation. This literature review, The authors are grateful to Sarah Thirlwall, Director of Strategic
up-to-date at the time of writing, has described these factors ICT Transformation, Counties Manukau Health, for comments on a
and referenced all key sources.1 Therefore, it provides a working draft of this article.
resource for those involved in planning and evaluating large HIS
evaluations. The factors identified in this literature review cross the Appendix A. Search strategy
socio-technical dimensions and include ensuring that there is suffi-
cient funding to purchase the appropriate technology and support Systematic search strategy in medical and relevant social science
its implementation through the entire chronology of the process. databases:
Strong ‘top-down’ leadership guidance and support is required and
a crucial aspect of this leadership is effective communication across Searched in the databases below for Search terms:
managerial, information technology, administrative, and clinical Transformat* and healthcare or hospital
boundaries. A ‘champion’ of the HIS can personify this communica- Hospital or healthcare and IT or IS or technolog* or program*
tion and ensure that end users are kept informed of the progress of Items were excluded if they were published before 2000,
the implementation. End users should be involved throughout all focused on specific technological devices, were duplicates of items
aspects of the implementation and need to know the potential ben- found in other databases, were not complex or transformational
efits so that they are prepared to tolerate the inevitable disruption initiatives
of the implementation. Sufficient time and adequate resourcing for 1. Emerald:
quality information technology training and support is also crucial. http://www.emeraldinsight.com/
End users need to have confidence that they can use the technol- 566 results for “Transformat* and technology and healthcare”
ogy and that it will work effectively. When individuals within the and got 326 results. 62 were included.
organisation understand the possible benefits to care and are aware 222 results for “hospital and technolog and program**”. 23 were
that all efforts have been taken to mitigate potential risks, they included after reviewing abstracts.
are more likely to support and use the HIS. A vital element in this 2. Factiva
process is having quality, easy to use technology which improves https://global.factiva.com/
the way that people deliver healthcare. Robust evaluation of HIS 88,428 results for “Transformat* and healthcare” which seem
implementation is required at all stages to provide ongoing feed- to be mostly articles by journalists (newspapers, online). Filtered
back. Perhaps the most important awareness for all those involved by subject (health) and industry (hospital care & computing) = 96
in the implementation of HIS is that it is complex and requires a results. These are nearly all press releases or short articles. Included
16.
3. Google Scholar
1
The recent book by Ammenworth and Rigby [172], published after the time
https://scholar.google.co.nz/
period of this review, is another key source/resource for planning and evaluating 36,100 results for “Transformation and healthcare” Added tech-
HIS. nology to search and took it back to 336,000. Refined search to
94 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97

post 2000, which gave 19,100 results. Searched and reviewed first A total of 382 publications was reviewed for this review.
500 entries ceasing at patents, non-refereed journals and foreign
language articles. Included 31 articles.
Appendix B. Key references for factors that promoting and
4a. Medline (via proquest)
inhibiting successful HIS implementation
http://search.proquest.com/medline/medline/advanced
Searched for ‘transformat* and “healthcare or hospital” from
Key References for Factors that Promote or Inhibit Successful
2000 to 2016. Got 15,895 results so searched within them for
HIS Implementation
‘technologic*’ and got 65 results and ‘information technolog* trans-
formation’ and hospital’. Reviewed 168 and included 28 articles. Promoting Factor References

4b. Medline (via ovid) Organisational and Currie and Finnegan [16], Doherty et al.
http://ovidsp.tx.ovid.com/ technological structures in [80], Fleuren [140], Harrison and
place Kimani [135], Hunter et al. [139],
Searched for ‘hospital and transformat*’ and limited search to
Robert et al. [30], Takian [87]
post 2000 in the categories of health technology assessment and Ongoing Evaluation Abraham and Jungla [77], Black et al.
health administration journals and got 1891 results. Used the tree throughout implementation [6], Brender [110], Nykänen et al.
for information systems and branch of information sciences which process [127], Oates et al. [111], Takian [87],
Wyatt and Wyatt [12]
includes category of information systems, which has a subcate-
Communication across the Anderson and Stafford [138], Cresswell
gory integrated advanced information systems. 288 results were organisation et al. [74], Potts et al. [23], Waterson
reviewed and 11 included. et al. [145]
5. Proquest Strong leadership Bernstein et al. [134], Degeling and
http://search.proquest.com/health/advanced?accountid=14700 Carr [130], Doolan et al. [84]Erskine
et al. [132], White et al. [133]
Searched for ‘information and technolog* and transformat*’ and
Adequate resourcing, including Doolan et al. [84], Li et al. [5], Morrow
‘healthcare’ or ‘hospital’ and got 111,197 results so added com- time/support for training et al. [75], Robert et al. [30], Sharma
plex to ‘information technology transformation’ and got 42,434 et al. [117]
results. Limited to hospitals OR health care OR health care indus- User involvement at all stages Kushniruk et al. [153], Li et al. [5],
try OR information technology OR health care management and of implementation Sherer et al. [152], Takian [87],
Themistocleous and Morobito [54], van
got 5446 results. Proquest has several databases so can limit which Gemert-Pijnen et al. [13]
it’s accessing – Proquest Health Management had 2638 results and End users understand Cresswell and Sheikh [10], Ludwick
Proquest Health and Medical Complete had 1163. Sorted by sub- perceived benefits of HIS and Doucette [144], Rye and Kimberley
ject to start with hospitals and include articles about hospitals that [31]
IT is fit for purpose Goldzweig et al. [26], Morrison et al.
had subcategories organisational change, innovations, case stud-
[167], Takian [87], Yusof et al. [128]
ies, technological change, hospital information systems and got 425 ‘Champion’ of the technology is Gagnon et al. [3], Pare et al. [148],
results. Reviewed and included 29. involved Udechukwu et al. [147]
6. Pubmed
http://www.ncbi.nlm.nih.gov/pubmed Inhibiting Factor References

Searched for ‘hospital’ and ‘information technology’ (which gave User resistance Gagnon [3], Hendy et al. [169], Rivard
option of ‘information technology system’ in indexing so used that) and LaPointe [170], Takian et al. [168]
Poor quality technology Ancker et al. [165], Lorenzi and Riley
and evaluation studies (550 results). Reviewed first 400 sorted by
[162], Powell-Cope et al. [166]
relevance which were primarily about evaluations of specific IT Organisational inflexibility Avison [21], Ellingsen and Monteiro
interventions. Included 13 articles. and/or instability [57], Harrison et al. [92], Kaplan and
7. Scopus Salamone [163]
https://www.scopus.com/ Lack of ‘fit’ between social, Ammenwerth et al. [129], Cresswell
technological and and Sheikh [10], Tsiknakis and
Searched information AND technology AND hospital. Refined by
organisational domains Kouroubali [39] Robert [30]
keyword – ‘hospital information systems’ and got 1960. Searched
within these for ‘transformation’ and got 60 results; ‘complex’ and
‘technolog*’ and got 211. Reviewed both of these searches. Included References
39.
8. SSRN [1] R. Hillestad, et al., Can electronic medical record systems transform health
care? Potential health benefits, savings, and costs, Health Aff. (Millwood) 24
http://papers.ssrn.com/sol3/DisplayAbstractSearch.cfm (5) (2005) 1103–1117.
Searched for ‘hospital’ and ‘information technolog*’ across the [2] E. Ammenwerth, et al., Evaluation of health information systems—problems
two research areas Health Economics, Information Systems & and challenges, Int. J. Med. Inf. 71 (2–3) (2003) 125–135.
[3] M.-P. Gagnon, et al., Systematic review of factors influencing the adoption of
eBusiness. Got 69 results. Reviewed and included 2.
information and communication technologies by healthcare professionals, J.
Web of Science Med. Syst. 36 (2012) 241–277.
http://apps.webofknowledge.com.ezproxy.otago.ac.nz/ [4] L. Lapointe, M. Mignerat, I. Vedel, The IT productivity paradox in health: a
stakeholder’s perspective, Int. J. Med. Inf. 80 (2) (2011) 102–115.
Search for ‘hospital’ and “information technology” gave 1751
[5] J. Li, A. Talaei-Khoei, H. Seale, P. Ray, C. MacIntyre, Health care provider
results. Refining by keyword ‘transformation’ gave 39 results. adoption of eHealth: systematic literature review, Interact. J. Med. Res. 2 (1)
Included 8. (2013).
Search for ‘complex’ within ‘hospital’ and “information technol- [6] A.D. Black, et al., The impact of eHealth on the quality and safety of health
care: a systematic overview, PLoS Med. 8 (1) (2011) e1000387.
ogy” results and got 132 results. Reviewed and included 13. [7] P. Jennett, et al., The socio-economic impact of telehealth: a systematic
Two further separate searches were also conducted. The first review, J. Telemed. Telecare 9 (6) (2003) 311–320.
focuses on identifying the literature about evaluation methods [8] B.-T. Karsh, et al., Health information technology: fallacies and sober
realities, J. Am. Med. Inform. Assoc. 17 (6) (2010) 617–623.
of transformational HIS. This garnered 21 relevant articles. The [9] A. de Bont, R. Bal, Telemedicine in interdisciplinary work practices: on an IT
second search was to identify systematic reviews of HIS imple- system that met the criteria for success set out by its sponsors, yet failed to
mentation articles. Fourteen relevant systematic reviews were become part of every-day clinical routines, BMC Med. Inform. Decis. Mak. 8
(2008) 47.
included. [10] K. Cresswell, A. Sheikh, Organizational issues in the implementation and
A ‘meta-narrative review’ as described in the methods section adoption of health information technology innovations: an interpretative
of the article obtained 57 further relevant publications. review, Int. J. Med. Inf. 82 (5) (2013) e73–e86.
 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97 95

[11] A.K. Jha, et al., The use of health information technology in seven nations, [43] S. Sood, et al., What is telemedicine? A collection of 104 peer-reviewed
Int. J. Med. Inf. 77 (12) (2008) 848–854. perspectives and theoretical underpinnings, Telemed. e-Health 13 (5)
[12] J.C. Wyatt, S.M. Wyatt, When and how to evaluate health information (2007) 573–590.
systems? Int. J. Med. Inf. 69 (2–3) (2003) 251–259. [44] T. Greenhalgh, et al., The organising vision for telehealth and telecare:
[13] J.E.W.C. van Gemert-Pijnen, et al., A holistic framework to improve the discourse analysis, BMJ Open 2 (4) (2012) e001574.
uptake and impact of eHealth technologies, J. Med. Internet Res. 13 (4) [45] M. McCartney, Show us the evidence for telehealth, BMJ 344 (2012).
(2011) e111. [46] A. Steventon, et al., Effect of telehealth on use of secondary care and
[14] C.L. Anderson, R. Agarwal, The digitization of healthcare: boundary risks, mortality: findings from the Whole System Demonstrator cluster
emotion, and consumer willingness to disclose personal health information, randomised trial, BMJ 344 (2012).
Inform. Syst. Res. 22 (3) (2011) 469–490. [47] F. Fatehi, R. Wootton, Telemedicine, telehealth or e-health? A bibliometric
[15] N. Barber, T. Cornford, E. Klecun, Qualitative evaluation of an electronic analysis of the trends in the use of these terms, J. Telemed. Telecare 18 (8)
prescribing and administration system, Qual. Saf. Health Care 16 (4) (2007) (2012) 460–464.
271–278. [48] T.R. Campion, C.S. Gadd, Peers, regulators, and professions: the influence of
[16] W.L. Currie, D.J. Finnegan, The policy-practice nexus of electronic health organizations in health information technology adoption, AMIA. Annual
records adoption in the UK NHS: an institutional analysis, J. Enterp. Inform. Symposium Proceedings/AMIA Symposium. AMIA Symposium, 2010 (2010)
Manage. 24 (2) (2013) 146–170. 86–90.
[17] R. Heeks, Health information systems: failure, success and improvisation, [49] M.H. Gabriel, et al., Progress and challenges: implementation and use of
Int. J. Med. Inf. 75 (2) (2006) 125–137. health information technology among critical-access hospitals, Health Aff.
[18] P. Littlejohns, J.C. Wyatt, L. Garvican, Evaluating computerised health (Millwood) 33 (7) (2014) 1262–1270.
information systems: hard lessons still to be learnt, BMJ 326 (7394) (2003) [50] A. Georgiou, J.I. Westbrook, J. Braithwaite, An empirically-derived approach
860–863. for investigating health information technology: the elementally entangled
[19] G. Southon, C. Sauer, K. Dampney, Lessons from a failed information systems organisational communication (EEOC) framework, BMC Med. Inform. Decis.
initiative: issues for complex organisations, Int. J. Med. Inf. 55 (1) (1999) Mak. 12 (1) (2012).
33–46. [51] D. Lewis, et al., Understanding the role of technology in health information
[20] M. Berg, Implementing information systems in health care organizations: systems, Pac. Health Dialog. 18 (1) (2012) 144–154.
myths and challenges, Int. J. Med. Inf. 64 (2–3) (2001) 143–156. [52] J.D. Restuccia, et al., Hospital implementation of health information
[21] D. Avison, T. Young, Time to rethink health care and ICT? Commun. ACM 50 technology and quality of care: are they related? BMC Med. Inform. Decis.
(6) (2007) 69–74. Mak. 12 (1) (2012) p109.
[22] B. Kaplan, Evaluating informatics applications—some alternative [53] C.J. Stokes, The electronic health revolution: how health information
approaches: theory, social interactionism, and call for methodological technology is changing medicine—and the obstacles in its way, Health Law
pluralism, Int. J. Med. Inf. 64 (1) (2001) 39–56. Policy Brief (2012), forthcoming.
[23] H.W.W. Potts, et al., Towards a Better Understanding of Delivering E-Health [54] M. Themistocleous, V. Morabito, How can user-centred design affect the
Systems: A Systematic Review Using The Meta-Narrative Method and Two acceptance and adoption of service oriented healthcare information
Case Studies. Final Report, NIHR Service Delivery and Organisation systems? Int. J. Healthcare Technol. Manage. 13 (5–6) (2012) 321–344.
Programme, London, England, 2011. [55] P. Waterson, Health information technology and sociotechnical systems: a
[24] M.B. Buntin, et al., The benefits of health information technology: a review progress report on recent developments within the UK National Health
of the recent literature shows predominantly positive results, Health Aff. Service (NHS), Appl. Ergon. 45 (2 PA) (2014) 150–161.
(Millwood) 30 (3) (2011) 464–471. [56] C. AbouZahr, T. Boerma, Health information systems: the foundations of
[25] B. Chaudhry, et al., Systematic review: impact of health information public health, Bull. World Health Organ. 83 (8) (2005) 578–583.
technology on quality, efficiency, and costs of medical care, Ann. Intern. [57] G. Ellingsen, E. Monteiro, The organizing vision of integrated health
Med. 144 (10) (2006) 742–752. information systems, Health Inform. J. 14 (3) (2008) 223–236.
[26] C.L. Goldzweig, et al., Costs and benefits of health information technology: [58] R.G. Fichman, R. Kohli, R. Krishnan, Editorial overview-the role of
new trends from the literature, Health Aff. (Millwood) 28 (2) (2009) information systems in healthcare: current research and future trends,
w282–w293. Inform. Syst. Res. 22 (3) (2011) 419–428.
[27] W.R. Hersh, et al., Clinical outcomes resulting from telemedicine [59] R. Haux, Health information systems–past, present, future, Int. J. Med. Inf. 75
interventions: a systematic review, BMC Med. Inform. Decis. Mak. 1 (1) (3–4) (2006) 268–281.
(2001) 1–8. [60] J.S. McCullough, The adoption of hospital information systems, Health Econ.
[28] S. McLean, et al., The impact of telehealthcare on the quality and safety of 17 (5) (2008) 649–664.
care: a systematic overview, PLoS One 8 (8) (2013) e71238. [61] P.W. O’Carroll, et al., Public Health Informatics and Information Systems,
[29] H. Mistry, Systematic review of studies of the cost-effectiveness of Springer-Verlag New York, Inc., New York, 2002.
telemedicine and telecare: changes in the economic evidence over twenty [62] M. Tsiknakis, D.G. Katehakis, S.C. Orphanoudakis, An open,
years, J. Telemed. Telecare 18 (1) (2012) 1–6. component-based information infrastructure for integrated health
[30] Robert, G., et al., Organisational factors influencing technology adoption and information networks, Int. J. Med. Inf. 68 (1–3) (2002) 3–26.
assimilation in the NHS: a systematic literature review, in Report for the [63] M. Jirotka, et al., Collaboration and trust in healthcare innovation: the
National Institute for Health Research Service Delivery and Organisation eDiaMoND case study, Comput. Support. Cooper. Work (CSCW) 14 (4)
programme 2009. (2005) 369–398.
[31] C.B. Rye, J.R. Kimberly, The adoption of innovations by provider [64] C. May, Agency and implementation: understanding the embedding of
organizations in health care, Med. Care Res. Rev. 64 (3) (2007) 235–278. healthcare innovations in practice, Social Sci. Med. 78 (2013) 26–33.
[32] P.G. Shekelle, S.C. Morton, E.B. Keeler, Costs and benefits of health [65] J.P. Bigus, et al., Information technology for healthcare transformation, IBM
information technology, Evid. Rep. Technol. Assess. 132 (2006) 1–71. J. Res. Dev. 55 (5) (2011).
[33] P.S. Whitten, et al., Systematic review of cost effectiveness studies of [66] K. Lynch, et al., The health IT regional extension center program: evolution
telemedicine interventions, BMJ 324 (7351) (2002) 1434–1437. and lessons for health care transformation, Health Serv. Res. 49 (1 pt 2)
[34] N.A.D. Connell, T.P. Young, Evaluating healthcare information systems (2014) 421–437.
through an enterprise perspective, Inform. Manage. 44 (4) (2007) 433–440. [67] V.P. Aggelidis, P.D. Chatzoglou, Methods for evaluating hospital information
[35] A.W. Kushniruk, et al., Emerging approaches to usability evaluation of systems: a literature review, EuroMed J. Bus. 3 (1) (2008) 99–118.
health information systems: towards in-situ analysis of complex healthcare [68] M. Farzandipour, F. Sadoughi, Z. Meidani, Hospital information systems user
systems and environments, in: Studies in Health Technology and needs analysis: a vendor survey, J. Health Inform. Dev. Countries 5 (1)
informatics, 2011. (2011).
[36] J. Car, et al., The impact of eHealth on the quality and safety of healthcare A [69] E.W.P. Ford, et al., Hospital IT adoption strategies associated with
Systemic Overview & Synthesis of the Literature Report for the NHS implementation success: implications for achieving meaningful
Connecting for Health Evaluation Programme 2008. use/PRACTITIONER APPLICATION, J. Healthc. Manag. 55 (3) (2010) 175–188,
[37] European Commission, I.a.M, eHealth in Action, Good Practice in European discussion 188–189.
Countries, European Commission, Luxembourg, 2009. [70] H.W. Lee, T. Ramayah, N. Zakaria, External factors in hospital information
[38] A.H. Goroll, et al., Community-wide implementation of health information system (HIS) adoption model: a case on Malaysia, J. Med. Syst. 36 (4) (2012)
technology: the Massachusetts eHealth collaborative experience, J. Am. 2129–2140.
Med. Inform. Assoc. 16 (1) (2009) 132–139. [71] M.M. Yusof, et al., Investigating evaluation frameworks for health
[39] M. Tsiknakis, A. Kouroubali, Organizational factors affecting successful information systems, Int. J. Med. Inf. 77 (6) (2008) 377–385.
adoption of innovative eHealth services: a case study employing the FITT [72] World Health Organisation, eHealth at WHO, Available from: http://www.
framework, Int. J. Med. Inf. 78 (1) (2009) 39–52. who.int/ehealth/about/en/.
[40] L. Van Velsen, J. Wentzel, J.E. Van Gemert-Pijnen, Designing eHealth that [73] G. Eysenbach, What is e-health? J. Med. Internet Res. 3 (2) (2001) pe20.
matters via a multidisciplinary requirements development approach, JMIR [74] K.M. Cresswell, A.A. Robertson Sheikh, Lessons learned from England’s
Res. Protoc. 2 (1) (2013) e21. national electronic health record implementation: implications for the
[41] C. Pagliari, et al., What is eHealth (4): a scoping exercise to map the field, J. international community, Proceedings of the 2nd ACM SIGHIT International
Med. Internet Res. 7 (1) (2005) e9. Health Informatics Symposium (2012), ACM.
[42] P.M. Yellowlees, Successfully developing a telemedicine system, J. Telemed. [75] E. Morrow, et al., Implementing large-scale quality improvement, Int. J.
Telecare 11 (7) (2005) 331–335. Health Care Qual. Assur. 25 (4) (2012) 237–253.
96 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97

[76] A. Steventon, et al., Effect of telehealth on use of secondary care and [107] C. Johnson, Why did that happen? Exploring the proliferation of barely
mortality: findings from the Whole System Demonstrator cluster usable software in healthcare systems, Qual. Saf. Health Care 15 (Suppl. 1)
randomised trial, BMJ 344 (2012). (2006) i76–i81.
[77] C. Abraham, I. Junglas, From cacophony to harmony: a case study about the [108] D. Blumenthal, M. Tavenner, The meaningful use regulation for electronic
IS implementation process as an opportunity for organizational health records, New Engl. J. Med. 363 (6) (2010) 501–504.
transformation at Sentara Healthcare, J. Strateg. Inform. Syst. 20 (2) (2011) [109] A.K. Jha, Meaningful use of electronic health records: the road ahead, JAMA
177–197. 304 (15) (2010) 1709–1710.
[78] H.C. Noel, et al., Home telehealth reduces healthcare costs, Telemed. J. [110] J. Brender, Handbook of Evaluation Methods for Health Informatics,
E-Health 10 (2) (2004) 170–183. Academic Press, 2006.
[79] C. Hawn, Take two aspirin and tweet me in the morning: how Twitter, [111] B.J. Oates, D.W.H.M. Wainwright Edwards, Endless Bad Projects or
Facebook, and other social media are reshaping health care, Health Aff. Evidence-Based Practice? An Agenda for Action, in Grand Successes and
(Millwood) 28 (2) (2009) 361–368. Failures in IT, Public and Private Sectors, Springer, 2013, pp. 619–624.
[80] N.F. Doherty, C. Ashurst, J. Peppard, Factors affecting the successful [112] E. Ammenwerth, et al., Visions and strategies to improve evaluation of
realisation of benefits from systems development projects: findings from health information systems reflections and lessons based on the HIS-EVAL
three case studies, J. Inform. Technol. 27 (1) (2012) 1–16. workshop in Innsbruck, Int. J. Med. Inf. 73 (2004) 479–491.
[81] R. Gauld, S. Goldfinch, Dangerous Enthusiasms: E-Government, Computer [113] J. Talmon, et al., The VATAM guidelines, Int. J. Med. Inf. 56 (1–3) (1999)
Failure and Information Systems Development, Univ of Otago Pr, 2006, 2016. 107–115.
[82] D. Shpilberg, et al., Avoiding the alignment trap in IT, MIT Sloan Manage. [114] P.E. Plsek, T. Greenhalgh, The challenge of complexity in health care, Br.
Rev. 49 (1) (2007) 51. Med. J. 323 (7313) (2001) 625.
[83] J.M. Brokel, M.I. Harrison, Redesigning care processes using an electronic [115] A. Shiell, P. Hawe, L. Gold, Complex interventions or complex systems?
health record: a system’s experience, Joint Comm. J. Qual. Patient Saf./Joint Implications for health economic evaluation, BMJ: Br. Med. J. 336 (7656)
Comm. Resour. 35 (2) (2009) 82–92. (2008) 1281–1283.
[84] D.F. Doolan, D.W. Bates, B.C. James, The use of computers for clinical care: a [116] E. Murray, et al., Why is it difficult to implement e-health initiatives? A
case series of advanced US sites, J. Am. Med. Inform. Assoc. 10 (1) (2003) qualitative study, Implement. Sci. 6 (1) (2011).
94–107. [117] U. Sharma, J. Barnett, M. Clarke, Understanding Innovation Deployment and
[85] J.R. Guard, et al., Integrated advanced information management systems: a Evaluation in Healthcare: The Triality Framework, INTECH Open Access
twenty-year history at the University of Cincinnati, J. Med. Libr. Assoc. 92 (2) Publisher, 2012.
(2004) 171–178. [118] P. Whitten, C. Kingsley, J. Grigsby, Results of a meta-analysis of cost-benefit
[86] C. Kenney, Transforming Health Care: Virginia Mason Medical Center’s research: is this a question worth asking? J. Telemed. Telecare (2000) 6.
Pursuit of the Perfect Patient Experience, CRC Press, 2012. [119] H.C. Kaplan, et al., The influence of context on quality improvement success
[87] A. Takian, Envisioning electronic health record systems as change in health care: a systematic review of the literature, Milbank Q. 88 (4)
management: the experience of an English hospital joining the National (2010) 500–559.
Programme for Information Technology, Stud. Health Technol. Inform. 180 [120] M.W. Chiasson, E. Davidson, Pushing the contextual envelope: developing
(2012) 901–905. and diffusing IS theory for health information systems research, Inform. Org.
[88] J. Car, K. Huckvale, H. Hermens, Telehealth for long term conditions, BMJ 344 14 (3) (2004) 155–188.
(2012) pe4201. [121] T. Greenhalgh, et al., Introduction of shared electronic records: multi-site
[89] J.S. Ash, M. Berg, E. Coiera, Some unintended consequences of information case study using diffusion of innovation theory, BMJ 337 (2008).
technology in health care: the nature of patient care information [122] R.J. Holden, B.-T. Karsh, A theoretical model of health information
system-related errors, J. Am. Med. Inform. Assoc. 11 (2) (2004) 104–112. technology usage behaviour with implications for patient safety, Behav.
[90] M. Caudill-Slosberg, W.B. Weeks, Case study: identifying potential problems Inform. Technol. 28 (1) (2009) 21–38.
at the human/technical interface in complex clinical systems, Am. J. Med. [123] F. Macfarlane, et al., Achieving and sustaining profound institutional change
Qual. 20 (6) (2005) 353–357. in healthcare: case study using neo-institutional theory, Social Sci. Med. 80
[91] Y.Y. Han, et al., Unexpected increased mortality after implementation of a (2013) 10–18.
commercially sold computerized physician order entry system, Pediatrics [124] E. Murray, et al., Normalisation process theory: a framework for developing,
116 (6) (2005) 1506–1512. evaluating and implementing complex interventions, BMC Med. 8 (1) (2010)
[92] M.I. Harrison, R. Koppel, S. Bar-Lev, Unintended consequences of 1.
information technologies in health care—an interactive sociotechnical [125] S. Cho, A Contextualist Approach to Telehealth Innovations, Georgia State
analysis, J. Am. Med. Inform. Assoc. 14 (5) (2007) 542–549. University, Ann Arbor, 2007, pp. 164.
[93] R. Koppel, et al., Role of computerized physician order entry systems in [126] P. Craig, et al., Developing and evaluating complex interventions: the new
facilitating medication errors, JAMA 293 (10) (2005) 1197–1203. Medical Research Council guidance, BMJ 337 (2008).
[94] R.M. Wachter, Expected and unanticipated consequences of the quality and [127] P. Nykänen, et al., Guideline for good evaluation practice in health
information technology revolutions, JAMA 295 (23) (2006) 2780–2783. informatics (GEP-HI), Int. J. Med. Inf. 80 (12) (2011) 815–827.
[95] G. Ellingsen, E. Monteiro, et al., The slight surprise of integration, in: C. [128] M.M. Yusof, et al., An evaluation framework for Health Information Systems:
Sorensen (Ed.), Designing Ubiquitous Information Environments: human, organization and technology-fit factors (HOT-fit), Int. J. Med. Inf. 77
Socio-Technical Issues and Challenges, Springer Science & Business Media, (6) (2008) 386–398.
2005, pp. 261–274. [129] E. Ammenwerth, C. Iller, C. Mahler, IT-adoption and the interaction of task,
[96] M. Bloomrosen, et al., Anticipating and addressing the unintended technology and individuals: a fit framework and a case study, BMC Med.
consequences of health IT and policy: a report from the AMIA 2009 Health Inform. Decis. Mak. 6 (2006) p3.
Policy Meeting, J. Am. Med. Inform. Assoc. 18 (1) (2011) 82–90. [130] P. Degeling, A. Carr, Leadership for the systemization of health care: the
[97] R.J. Holden, Cognitive performance-altering effects of electronic medical unaddressed issue in health care reform, J. Health Organ Manag. 18 (6)
records: an application of the human factors paradigm for patient safety, (2004) 399–414.
Cognit. Technol. Work 13 (1) (2011) 11–29. [131] G. Dickson, Transformations in Canadian health systems leadership: an
[98] J.S. Ash, et al., The unintended consequences of computerized provider order analytical perspective, Leadersh. Health Serv. 22 (4) (2013) 292–305.
entry: findings from a mixed methods exploration, Int. J. Med. Inf. 78 (1) [132] J. Erskine, et al., Leadership and transformational change in healthcare
(2009) 69–76. organisations: a qualitative analysis of the North East Transformation
[99] E.M. Campbell, et al., Types of unintended consequences related to System, Health Serv. Manage. Res. 26 (1) (2013) 29–37.
computerized provider order entry, J. Am. Med. Inform. Assoc. 13 (5) (2006) [133] M. White, J. Wells, T. Butterworth, Leadership, a key element of quality
547–556. improvement in healthcare. Results from a literature review of Lean
[100] J.G. Anderson, Social, ethical and legal barriers to e-health, Int. J. Med. Inf. 76 Healthcare and the Productive Ward: releasing time to care initiative, Int. J.
(5) (2007) 480–483. Leadersh. Public Serv. 9 (3/4) (2011) 90–108.
[101] G. Fairbrother, et al., Cincinnati Beacon Community Program highlights [134] M.L. Bernstein, T. McCreless, M.J. Côté, Five constants of information
challenges and opportunities on the path to care transformation, Health technology adoption in healthcare, Hosp. Top. 85 (1) (2007) 17–25.
Affairs (Project Hope) 33 (5) (2014) 871–877. [135] M.I. Harrison, J. Kimani, Building capacity for a transformation initiative:
[102] M. Meadors, et al., Going live implementing an electronic health record system redesign at Denver Health, Health Care Manage. Rev. 34 (1) (2009)
system in the emergency department, Proceedings of the International 42–53.
Symposium on Human Factors and Ergonomics in Health Care (2014), SAGE [136] S. Szydlowski, C. Smith, Perspectives from nurse leaders and chief
Publications. information officers on health information technology implementation,
[103] C. May, A rational model for assessing and evaluating complex interventions Hosp. Top. 87 (1) (2009) 3–9.
in health care, BMC Health Serv. Res. 6 (1) (2006) 1. [137] F. Thomson, et al., Understanding the impact of information technology on
[104] S. Cho, L. Mathiassen, A. Nilsson, Contextual dynamics during health interactions between patients and healthcare professionals: the
information systems implementation: an event-based actor-network INTERACT-IT study, Health Serv. J. 2012 (Suppl) (2016) P46.
approach, Eur. J. Inform. Syst. 17 (6) (2008) 614–630. [138] L.K. Anderson, C.J. Stafford, The big bang implementation: not for the faint of
[105] R.H. Miller, I. Sim, Physicians’ use of electronic medical records: barriers and heart, Comput. Nurs. 20 (1) (2002) 14–20, quiz 20–22.
solutions, Health Aff. (Millwood) 23 (2) (2004) 116–126. [139] D.J. Hunter, et al., Doing transformational change in the English NHS in the
[106] P. Powers, E-health leads nova scotia’s healthcare transformation, Healthc. context of big bang redisorganisation: findings from the North East
Q. (Toronto, Ont.) 12 (4) (2009) 116–121. transformation system, J. Health Organ Manag. 29 (1) (2011) 10–24.
 J. Sligo et al. / International Journal of Medical Informatics 97 (2017) 86–97 97

[140] M. Fleuren, K. Wiefferink, T. Paulussen, Determinants of innovation within [157] J. Walker, et al., The value of health care information exchange and
health care organizations, Lit. Rev. Delphi study 16 (2) (2004) 107–123. interoperability, Health affairs 24 (2005) pW5.
[141] T. Greenhalgh, et al., Introducing a nationally shared electronic patient [158] K.B. Eden, et al., Barriers and facilitators to exchanging health information: a
record: case study comparison of Scotland, England,Wales and Northern systematic review, Int. J. Med. Inf. 88 (2016) 44–51.
Ireland, Int. J. Med. Inf. 82 (5) (2013) e125–e138. [159] W. Hersh, Health care information technology: progress and barriers, JAMA
[142] R. Thomas, et al., Information management and technology in England’s 292 (18) (2004) 2273–2274.
large acute NHS hospitals: national strategy versus local reality, J. Manag. [160] R.J. Holden, B.-T. Karsh, The Technology Acceptance Model: its past and its
Med. 9 (1) (1995) 40–49. future in health care, J. Biomed. Inform. 43 (1) (2010) 159–172.
[143] F. Doolan-Noble, et al., How well does your healthcare system perform: [161] E. Zainuddin, G. Bassellier, I. Benbasat, Outsourcing projects success: the
tracking progress toward the triple aim using system level measures, N. Z. role of competence and leadership of the vendors and clients project
Med. J. 128 (1415) (2014) 44–50. managers, Proceedings of the 2010 Special Interest Group on Management
[144] D.A. Ludwick, J. Doucette, Adopting electronic medical records in primary Information System’s 48th Annual Conference on Computer Personnel
care: lessons learned from health information systems implementation Research on Computer Personnel Research (2010), ACM.
experience in seven countries, Int. J. Med. Inf. 78 (1) (2009) 22–31. [162] N.M. Lorenzi, R.T. Riley, Organizational issues=change, Int. J. Med. Inf. 69
[145] P. Waterson, P.L.T. Hoonakker, P. Carayon, Special issue on human factors (2–3) (2003) 197–203.
and the implementation of health information technology (HIT): comparing [163] B. Kaplan, K.D. Harris-Salamone, Health IT success and failure:
approaches across nations, Int. J. Med. Inf. 82 (5) (2013) 277–280. recommendations from literature and an AMIA workshop, J. Am. Med.
[146] A. Geissbuhler, Lessons learned implementing a regional health information Inform. Assoc. 16 (3) (2009) 291–299.
exchange in Geneva as a pilot for the Swiss national eHealth strategy, Int. J. [164] M.P. Gagnon, et al., Telehealth adoption in hospitals: an organisational
Med. Inf. 82 (5) (2013) e118–e124. perspective, J. Health Organ Manag. 19 (1) (2005) 32–56.
[147] O. Udechukwu, et al., Softer perspectives on enhancing the patient [165] J.S. Ancker, et al., The Triangle Model for evaluating the effect of health
experience using IS/IT, Int. J. Health Care Qual. Assur. 23 (2) (2010) 187–208. information technology on healthcare quality and safety, J. Am. Med. Inform.
[148] G. Pare, et al., Prioritizing the risk factors influencing the success of clinical Assoc. 19 (1) (2012) 61.
information system projects, Methods Inf. Med. 47 (3) (2008) 251–259. [166] G. Powell-Cope, A.L. Nelson, E.S. Patterson, Patient care technology and
[149] M.d.C. Caccia-Bava, V.C.K. Guimaraes, T. Guimaraes, Testing some major safety, in: R. Hughes (Ed.), Patient Safety and Quality: An Evidence-based
determinants for hospital innovation success, Int. J. Health Care Qual. Assur. Handbook for Nurses, Agency for Healthcare Research and Quality, Rockville
22 (5) (2013) 454–470. MD, 2008.
[150] A. Takian, Envisioning electronic health record systems as change [167] Z. Morrison, Understanding the implementation and adoption of the NHS
management: the experience of an English hospital joining the National Care Records Service (CRS) in english secondary care, in looking back,
Programme for Information Technology, Stud. Health Technol. Inform. 180 moving forward: capturing lessons and building the evidence base for
(2012) 901–905. health informatics, in: The Connecting for Health Evaluation Programme,
[151] K. Keshavjee, et al., A complex adaptive systems perspective of health University of Birmingham, 2012.
information technology implementation, Stud. Health Technol. Inform. 183 [168] A. Takian, A. Sheikh, N. Barber, We are bitter, but we are better off: case
(2013) 209–213. study of the implementation of an electronic health record system into a
[152] S.A. Sherer, et al., Integrating commercial ambulatory electronic health mental health hospital in England, BMC Health Serv. Res. 12 (2012) 484.
records with hospital systems: an evolutionary process, Int. J. Med. Inf. 84 [169] J. Hendy, et al., Challenges to implementing the national programme for
(9) (2015) 683–693. information technology (NPfIT): a qualitative study, BMJ 331 (7512) (2005)
[153] A. Kushniruk, et al., Integrating technology-centric and user-centric system 331–336.
testing methods: ensuring healthcare system usability and safety, in: [170] S. Rivard, L. Lapointe, Information technology implementers’ responses to
Studies in Health Technology and Informatics, 2010. user resistance: nature and effects, MIS Q. 36 (3) (2012) 897–920.
[154] L. Politi, et al., Use patterns of health information exchange systems and [171] P. Waterson, A critical review of the systems approach within patient safety
admission decisions: reductionistic and configurational approaches, Int. J. research, Ergonomics 52 (10) (2009) 1185–1195.
Med. Inf. 84 (12) (2016) 1029–1038. [172] E. Ammenwerth, M. Rigby, Evidence-Based Health Informatics: Promoting
[155] D.J. Brailer, Interoperability: the key to the future health care system, Health Safety and Efficiency Through Scientific Methods and Ethical Policy, vol. 222,
affairs 24 (2005) W5. IOS Press, 2016.
[156] H. Park, et al., Can a health information exchange save healthcare costs?
Evidence from a pilot program in South Korea, Int. J. Med. Inf. 84 (9) (2015)
658–666.