Information and Software Technology 54 (2012) 239–247 Contents lists available at SciVerse ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof IT Service Management Process Improvement based on ISO/IEC 15504: A systematic review Antoni Lluís Mesquida a,⇑, Antonia Mas a, Esperança Amengual a, Jose A. Calvo-Manzano b a b Department of Mathematics and Computer Science, University of the Balearic Islands, Ctra. de Valldemossa, Km. 7.5, 07122 – Palma de Mallorca, Spain Faculty of Computer Science, Technical University of Madrid, Campus de Montegancedo s/n, 28660 – Boadilla del Monte, Madrid, Spain a r t i c l e i n f o Article history: Received 3 February 2010 Received in revised form 14 September 2011 Accepted 10 November 2011 Available online 20 November 2011 Keywords: Software Process Improvement (SPI) ISO/IEC 15504 (SPICE) IT Service Management (ITSM) Systematic review a b s t r a c t Context: In recent years, many software companies have considered Software Process Improvement (SPI) as essential for successful software development. These companies have also shown special interest in IT Service Management (ITSM). SPI standards have evolved to incorporate ITSM best practices. Objective: This paper presents a systematic literature review of ITSM Process Improvement initiatives based on the ISO/IEC 15504 standard for process assessment and improvement. Method: A systematic literature review based on the guidelines proposed by Kitchenham and the review protocol template developed by Biolchini et al. is performed. Results: Twenty-eight relevant studies related to ITSM Process Improvement have been found. From the analysis of these studies, nine different ITSM Process Improvement initiatives have been detected. Seven of these initiatives use ISO/IEC 15504 conformant process assessment methods. Conclusion: During the last decade, in order to satisfy the on-going demand of mature software development companies for assessing and improving ITSM processes, different models which use the measurement framework of ISO/IEC 15504 have been developed. However, it is still necessary to define a method with the necessary guidelines to implement both software development processes and ITSM processes reducing the amount of effort, especially because some processes of both categories are overlapped. Ó 2011 Elsevier B.V. All rights reserved. Contents 1. 2. 3. 4. 5. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ITSM standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Systematic review method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1. Question formularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. Selection of sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3. Selection of studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4. Information extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5. Results summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1. Classification of studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2. Results from the analysis of primary studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1. Standards used for ITSM Process Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2. New ITSM Process Improvement initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3. Temporal view of primary studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions and future work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix A. List of primary studies in the systematic review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1. List of primary studies in the systematic review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 240 240 241 241 241 242 242 242 244 244 244 244 245 245 246 246 246 247 ⇑ Corresponding author. E-mail addresses: antoni.mesquida@uib.es (A.L. Mesquida), antonia.mas@uib.es (A. Mas), eamengual@uib.es (E. Amengual), joseantonio.calvomanzano@upm.es (J.A. CalvoManzano). 0950-5849/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.infsof.2011.11.002 240 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 1. Introduction Over the course of the last two decades, both large and small software companies have made important efforts in order to implement Software Process Improvement (SPI) programmes to establish a process-oriented culture in the organization. A focus on processes provides the stability and the management infrastructure required to deal with the ever-changing and competitive market. A well-defined management framework results in a better monitoring of the processes so that they can reach higher maturity levels. Higher maturity enables a global understanding and a better vision of the workload and, as a result, a more consistent and repeatable work. Moreover, technical skills of the staff are improved and the use of technology is maximized. When the productivity and efficiency of development activities are improved, the organization can develop, maintain and deliver high quality products, meeting business objectives (usually focused on quality improvement and cost and time reduction) and obtaining a higher customer satisfaction. The two most internationally used SPI models are Capability Maturity Model Integration (CMMI) [1] and ISO/IEC 15504 (SPICE) [2]. These models define a process improvement approach that provide organizations with the essential elements to set process improvement goals, establish a point of reference for assessing current processes and support the improvement of their performance. Nowadays, customers not only demand quality products obtained from mature processes, but they also require quality in the services they receive. In recent years, while companies have been deploying their software development processes, there has been an on-going demand for better IT services. In order to satisfy this demand, SPI international standards have evolved and have been adapted to incorporate IT Service Management (ITSM) best practices. For mature software development companies, besides the SPI initiative in which they are involved, the implementation of an ITSM standard has currently become one of the main priorities to assure their continuity and maximize the return of investment and business opportunities. The main goal of our research is to analyse existent relations between two disciplines that have traditionally been treated separately: SPI and ITSM. With the intention of taking advantage of the experience of the authors in the application of the ISO/IEC 15504 standard [3–7], the scope of the research is focused on this SPI standard and pursues a double objective:  Facilitate the implementation of ITSM processes in software companies already involved in an ISO/IEC 15504 SPI initiative.  Maximize the efficiency of a joint implementation of ISO/IEC 15504 and an ITSM standard. More concretely, the intention is to consider ISO/IEC 20000 as the ITSM standard. The main reason for this decision is that both standards have been developed by the same organization and thus compatibility among them should be feasible. Moreover, some processes of both ISO standards are overlapped. As a first step of our research, a systematic review of the literature which deals with existing initiatives of ITSM Process Improvement initiatives based on ISO/IEC 15504 for process assessment and improvement is presented in this paper. 2. ITSM standards ITSM is a process-oriented discipline which combines process management and industry best practices into a standard approach for optimizing IT services. ITSM provides a framework to structure IT operations that enables organizations to deliver quality IT services to meet business needs and adhere to service level agreements. Because ITSM is process-focused, it shares common interests with the process improvement movement. ITSM provides specific processes, frameworks, methodologies and guidance to manage planning, implementation and assessment of IT service processes to optimize tactical and strategic IT operations-related activities. Some of the most internationally accepted ITSM standards are ITIL (Information Technology Infrastructure Library), ISO/IEC 20000 and CMMI for Services (CMMI-SVC). In spite of the diversity of fields that this work deals with, and with the intention of making its comprehension easier, this section offers a brief description of the different standards mentioned so far: ISO/IEC 15504, ITIL, ISO/IEC 20000, and CMMI-SVC. ISO/IEC 15504 Information Technology – Process Assessment [2] is an International Standard for process assessment and improvement. It can be used by any organization to determine the current and potential capability of its own processes, and also to define areas and priorities for process improvement. In order to perform a process assessment conformant with the assessment model defined in ISO/IEC 15504-2 [8], a Process Assessment Model (PAM) based upon a suitable Process Reference Model (PRM) needs to be properly defined. Regarding ITSM, a new part describing an ITSM PAM is currently under development, called ISO/IEC NP TR 15504-8 Information Technology – Software process assessment – Part 8: An exemplar process assessment model for IT Service Management [9]. ITIL is a series of documents that contains a set of best practices to aid the implementation of a framework for ITSM. ITIL has been accepted in the industry as the de facto standard for ITSM around the world. It is focused on the continual measurement and the quality improvement of the services, from both the business and the client perspectives. The current version of ITIL, ITIL V3, contains five reference books [10–14] and its main contribution is the definition of a structure based on the services life cycle. ISO/IEC 20000 is an ITSM quality standard that promotes the adoption of an integrated process approach to effectively deliver managed services to meet the business and customer requirements. This standard, consisting of two parts [15,16], is under a review process in order to provide a better alignment with ITIL V3 and other ISO standards. New part 4 [17] will describe an ITSM PRM which is expected to be closely related to the new Part 8 of ISO/IEC 15504 [9]. CMMI models are collections of best practices developed by the Software Engineering Institute (SEI) for the application of process improvement in the development of products and services covering the entire product lifecycle. Current CMMI version, CMMI V1.3, was created with the main intention of defining constellations for being applied in different areas of interest. Regarding to Service Management, the CMMI-SVC [18] constellation was particularly created for Service Management process improvement. CMMI-SVC includes a collection of services best practices that provides guidance for the application of CMMI-SVC by a service provider organization. CMMI-SVC best practices focus on activities for providing quality services to the customer and end users. 3. Systematic review method The research is undertaken as a systematic literature review based on the guidelines proposed by Kitchenham [19,20] and the review protocol template developed by Biolchini et al. [21] which describes each phase of the systematic review process in terms of template sections. Other systematic reviews in SPI [22–24] have also been used as a basis for the work presented in this paper. A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 The protocol used for the systematic review is composed of five different stages: Question formularization, selection of sources, selection of studies, information extraction and results summarization. These five stages are detailed in the next sections. 3.1. Question formularization With the aim of defining the context in which the systematic review is applied, the protocol suggests to specify a set of items [21]. In the particular case of our systematic review, each item has been defined specifically for ITSM Process Improvement initiatives based on ISO/IEC 15504 International Standard for process assessment and improvement.  Problem: software companies have shown a growing interest in providing quality services to their customers. SPI could be extended in order to include ITSM processes. The context of this research is particularly focused on ITSM Process Improvement models based on the ISO/IEC 15504 assessment framework.  Question: what ITSM Process Improvement initiatives based on ISO/IEC 15504 exist?  Keywords and synonyms: Software Process Improvement (SPI), IT Service Management (ITSM), ISO/IEC 15504 (SPICE).  Intervention: analyse ITSM Process Improvement according to ISO/IEC 15504.  Control: there are no initial data for this systematic review.  Effect: identify all the ITSM Process Improvement initiatives, frameworks and models performed according to ISO/IEC 15504.  Outcome measure: the number of identified initiatives, frameworks and models.  Population: the set of research proposals related to ITSM Process Improvement and ISO/IEC 15504 which have been published in the list of sources selected for conducting the systematic review.  Application: companies interested in assessing and improving their ITSM processes within an ISO/IEC 15504-based SPI initiative. Researchers working on SPI models, more concretely on ISO/IEC 15504, or on IT Service Management standards.  Experimental design: none statistical analysis methods will be applied. 3.2. Selection of sources To perform the selection of the sources where searches for primary studies will be executed, the systematic review protocol proposes to address the following issues: definition of source selection criteria, language of the studies, identification of sources, definition of search strings, and selection of sources [21]. With regard to source selection criteria, the following criteria have been defined: Publishing companies or websites suggested by experts. High-impact publications. Availability of search mechanisms using keywords. Non-variability in search results by using the same set of keywords.  Availability on the web.     Concerning language studies, the obtained primary studies must be written in English. The sources have been identified on the basis of the judgement of the authors of this paper. The list of sources includes relevant journals in which SPI research area is widely dealt with, such as: Information and Software Technology, Software Process: Improvement and Practice, IEEE Software, IEEE Transactions on Software Engineering and ACM SIGSOFT Software Engineering Notes, Software Quality 241 Journal, among others. Moreover, the authors have also considered papers published in the proceedings of different conferences, such as: the EuroSPI Conference where experts discuss and exchange SPI practices and the SPICE Conference which is one of the most relevant events regarding the ISO/IEC 15504 standard, among others. By taking the list of keywords defined in Section 3.1 and making combinations with the logical operators ‘‘AND’’ and ‘‘OR’’, the search strings shown in Table 1 have been obtained. To carry out the searches, these search strings need to be adapted to each of the search engines of the selected sources. Taking into account the defined sources selection criteria the initial list of sources is shown in Table 2. Each element of the initial list of sources presented above has been evaluated according to the source selection criteria. After this evaluation, all the elements have been included in the final list of sources. The authors of the paper have evaluated the list of sources obtained and approved all the elements of this list. 3.3. Selection of studies Once the sources are defined, it is necessary to describe the process and the criteria for studies selection and evaluation [21]. The criteria by which studies will be evaluated to decide if they must be selected or excluded in the context of the systematic review have been defined by the authors of the paper taking into account Kitchenham’s proposals [19,20]. These criteria, Inclusion Criteria (IC) and Exclusion Criteria (EC), are shown in Table 3. The process performed to obtain and evaluate primary studies according to the defined inclusion and exclusion criteria is illustrated as a flow diagram in Fig. 1. This flow diagram shows two main groups of activities. The goal of the first group is the selection of primary studies. The second group of activities aims to extract the information of the selected primary studies. Information extraction will be presented later in this paper in Section 3.4. With regard to the selection of primary studies, the analysis of the title and the keywords will be the main inclusion criteria. In case this information is not enough to decide about the inclusion or the exclusion of the study then the abstract will be also analysed and the full text, if necessary. Initially all types of primary studies related to ITSM Process Improvement according to ISO/IEC 15504 will be taken into account. More concretely, the focus will be on studies presenting ITSM Process Improvement models based on the ISO/IEC 15504 assessment framework and the results of the application of these models to software companies. Table 4 shows the distribution of the studies obtained from each search source. The format of this table is adapted from other systematic reviews in software engineering [22–24]. As a result of the search execution 1944 studies have been obtained for further evaluation. Table 4 shows the number of initial studies obtained from each source (see the column ‘‘Discovered’’). After applying the inclusion criteria IC1, IC2, IC3 and IC4, defined in Table 3, only 82 of the 1944 discovered articles have been considered as relevant articles. Applying the criterion EC2 for the exclusion of duplicated articles, only 49 articles have been obtained. From these, applying the criterion EC1, finally 28 articles have been selected as primary studies. These results are shown in Table 1 Search strings. Search strings 1 2 3 (15504 OR SPICE) AND (ITSM OR ‘‘IT service’’ OR ‘‘service’’) (ITSM OR ‘‘IT service’’ OR ‘‘service’’) AND (improvement OR assessment) AND (15504 OR SPICE) (15504 OR SPICE) AND ‘‘service’’ AND (ITIL OR 20000 OR ‘‘CMMI-SVC’’) 242 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247  Study problems: study limitations and the deviation between the expected and the obtained results. Table 2 List of sources. Source 1 2 3 4 5 6 7 8 9 10 11 12 Name Web site ACM Portal (Digital Library & Guide) IEEE Computer Society Digital Library IEEE Xplore Springer Link ScienceDirect Wiley InterScience http://portal.acm.org/portal.cfm CiteSeerX IET Digital Library ISI Web of Knowledge Google Scholar EuroSPI (Proceedings of) (since 2004) SPICE (Proceedings of) (since 2003) http://www.computer.org/portal/ web/csdl http://ieeexplore.ieee.org http://www.springerlink.com http://www.sciencedirect.com http:// www.interscience.wiley.com http://citeseerx.ist.psu.edu http://www.ietdl.org http://www.isiknowledge.com http://scholar.google.com http://www.eurospi.net http://www.spiceconference.com The second category, subjective results, refers to those results that cannot be extracted directly from the study. The ‘‘Results Extraction’’ section of the form shows the study outcomes, the study contribution, as well as other issues. After a complete and detailed reading of each primary study and an unbiased evaluation of the information, the objective and subjective results have been extracted. During the information extraction execution, the authors’ perception of some aspects varied. Nevertheless, none of them was considered an important divergence. The authors reached a consensus and different perceptions were considered complementary to obtain a comprehensive analysis of the study. 3.5. Results summarization The last stage of the systematic review protocol aims to present the data resulting from the selected primary studies [21]. the last row of Table 4. The complete list of selected primary studies is presented in Appendix A of this article. This primary study selection has been reviewed by the authors in order to guarantee the quality of the included studies. 3.4. Information extraction Once primary studies are selected, the extraction of relevant information begins [21]. The criteria by which the information obtained from the studies has been included were defined. These Information Inclusion Criteria (ICinf) are presented in Table 5. To analyze the data obtained from the selected primary studies and to standardize the way in which information should be registered, a data extraction form has been designed in order to meet our particular research goals. This form, which is shown in Table 6, has been used to record comments, impressions and the most important ideas from each primary study. The structure and contents of this form are based on the information extraction format proposed in [25] and other systematic reviews [22–24]. The contents are grouped into objective and subjective results. The first category, objective results, refers to those results that can be directly extracted from the selected primary studies. These results have been organized in the form into the following four sections:  Study identification: the main data to identify the primary study: a consecutive number assigned to the paper, the publication title, its authors, the author’s contact information, the journal or conference in which it was published, the date of publication and the source from which it was obtained.  Study methodology: the type of the study, the country in which it was developed or applied, the improvement strategy goals and the ITSM Process Improvement models used or developed, among other information.  Study results: information about the improved processes, the key factors for successful improvement and the conclusions.  Results statistical calculus: with the purpose of showing the relevant findings of this systematic review, a statistical analysis has been conducted with the information extracted from the selected primary studies. A more detailed discussion of these results is further presented in Section 4 of this article.  Results presentation in tables: with the aim of facilitating the analysis, the results obtained from the systematic review have been summarized in tables and displayed in graphics under different perspectives. These results are shown in Tables 7–10 and in Fig. 2 in Section 4.  Sensitivity analysis: it was not applied.  Plotting: it was not applied.  Final comments: – Number of studies: 1944 studies were found and 28 of them were considered primary studies. – Search, selection and extraction bias: none search, selection or information extraction biases that could invalidate the systematic review results were identified. – Publication bias: none was defined. – Inter-reviewers variation: there were no divergences between reviewers regarding the systematic review results. – Results application: the obtained results suggest that new ITSM Process Improvement models have arisen to satisfy the growing demand of software companies for managing the services they provide. – Recommendations: the results of the systematic review presented in this paper should be applied to get an idea of the state of the art of ITSM Process Improvement based on the ISO/IEC 15504 standard. 4. Results and discussion This section presents a summary of the results after the systematic review execution. Table 3 Definition of studies inclusion and exclusion criteria. Criterion Description IC1 IC2 IC3 IC4 EC1 EC2 Include papers whose title is related to ITSM Process Improvement according to ISO/IEC 15504 Include papers that contain keywords that match with those defined in the search string Include papers whose abstract is related to the topic under consideration Include papers that contain information related to the definition or application of ITSM Process Improvement models Exclude those papers that refer to ITSM Process Improvement and ISO/IEC 15504 separately, without showing any kind of relationship between both topics Exclude all duplicated papers 243 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 Fig. 1. Procedure for executing the systematic review. Table 4 Distribution of studies by source. Source Search date ACM Portal IEEE Computer Society Digital Library IEEE Xplore Springer Link ScienceDirect Wiley InterScience CiteSeerX IET Digital Library ISI Web of Knowledge Google Scholar SPICE EuroSPI 25/07/2011 25/07/2011 25/07/2011 26/07/2011 25/07/2011 25/07/2011 26/07/2011 26/07/2011 26/07/2011 26/07/2011 27/07/2011 27/07/2011 Total Discovered Relevant Not repeated 15 98 43 178 106 11 99 0 39 778 211 366 0 3 2 9 0 1 2 0 4 42 8 11 0 2 0 3 0 0 1 0 2 30 4 7 Primaries 0 2 0 3 0 0 1 0 2 9 4 7 % 0 7 0 11 0 0 4 0 7 32 14 25 1944 82 49 28 100 Table 5 Definition of information inclusion criteria. Criterion Description IC1inf IC2inf IC3inf IC4inf Identify existent methodologies, techniques, methods and procedures for ITSM Process Improvement Collect information about proposed initiatives or models used to assess or improve ITSM processes Collect information about ITSM Process Improvement strategies followed by software organizations Collect information about improved processes and key factors for successful ITSM Process Improvement in software organizations 244 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 4.2.1. Standards used for ITSM Process Improvement Table 8 presents a classification of the primary studies regarding each different standard used for ITSM Process Improvement. Regarding process reference models, the results from the analysis of the primary studies reveal that ISO/IEC 20000 is the ITSM process reference model most frequently used. ITIL V2 and ITIL V3 follow in order of importance. ITIL V1, which is out of market, is only mentioned in two primary studies. Moreover, ISO new work item proposals, specifically ISO/IEC 15504-8 and ISO/IEC 20000-4, define an ITSM process assessment model and an ITSM process reference model, respectively. Finally, CMMI models, including the CMMI-SVC constellation, have also been used for guidance in ITSM Process Improvement initiatives. Table 6 Information extraction form. Information extraction form Study identification Consecutive number: Publication title: Authors: Contact information: Journal/Conference: Date: Source: Study methodology Type of Study: Country: Improvement strategy goals: Models/Standards used: Models/Standards developed: Study results Improved processes: Improvement success key factors: Conclusions: Study problems Limitations: Deviation between expected and obtained results: Results extraction Study outcomes: Study contribution: Other issues: 4.1. Classification of studies After the extraction of relevant information from each primary study, it was possible to determine that these studies could be classified into the four categories presented in Table 7. According to Table 7, most of the studies provide a new ISTM Process Improvement model based on ISO/IEC 15504. In addition, there is an interest in analysing the need of ITSM Processes Improvement, as well as combining ISO/IEC 15504 and ITSM Process Improvement standards. Finally, five primary studies offer a summary of the results of the application of new ITSM Process Improvement models. 4.2. Results from the analysis of primary studies This section details the results obtained after analysing the primary studies. These results have been classified into two different groups. 4.2.2. New ITSM Process Improvement initiatives Table 9 presents the initiatives specifically created with the aim of developing new ITSM Process Improvement models. Table 10 shows the distribution of these new ITSM Process Improvement models depending on the process assessment method and the process reference model they use. As Table 10 shows, both SCAMPI and ISO/IEC 15504-2 have been used as process assessment methods. Then, the feasibility of using these measurement frameworks for ITSM process assessment and improvement has already been tested. Starting with the first row, MITO and IT Service CMM are two ITSM Process Improvement initiatives with self-developed process reference models which use SCAMPI as assessment method. Regarding ITSM initiatives based on ISO/IEC 15504, the assessment method defined in ISO/IEC 15504 Part 2 has been used to assess both ITIL and ISO/IEC 20000 processes, as well as to assess new developed process reference models:  SPINI+ defines an ITSM process library from ITIL and ISO/IEC 20000.  The TIPA, NOEMI and SPICE Lite [ITSM] models use ITIL (V2) as process reference model.  TickIT Plus core includes, among others, ISO/IEC 20000 as optional standard in its certification model.  SPICE 1-2-1 for ISO 20000 uses the ISO/IEC 15504-2 assessment method for ISO/IEC 20000 process assessment.  The NiCE model uses the ISO/IEC 15504-2 assessment method to assess the processes provided by NOVE-IT, a set of IT procurement, development, operation and service provision processes. Table 7 Classification of primary studies. Category Primary studies Percentage of primary studies 1 2 3 7, 8, 14, 17, 20, 25 2, 5, 12, 15, 16, 24 1, 3, 4, 6, 9, 11, 18, 19, 21, 22, 28 21.4% (6/28) 21.4% (6/28) 39.3% (11/28) 10, 13, 23, 26, 27 17.9% (5/28) 4 Analysis of the need of assessing and improving ITSM processes. Combining ISO/IEC 15504 and ITSM Process Improvement frameworks. Development of an ISO/IEC 15504-based model, framework or approach for ITSM Process Improvement. Results of the application of an ITSM Process Improvement model. Table 8 Standards used for ITSM. Standard 1 2 3 4 5 6 7 ISO/IEC 20000 ITIL V1 ITIL V2 ITIL V3 ISO/IEC 15504–8 & 20000–4 CMM/CMMI CMMI-SVC Primary studies Percentage of primary studies 1, 3, 4, 5, 7, 8, 12, 14, 15, 16, 18, 19, 21, 22, 25, 26, 28 20, 25 1, 2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 15, 22, 26, 27 4, 8, 11, 17, 19, 23, 27, 28 4, 5, 7, 8, 19, 21, 24, 26 7, 11, 16, 17, 20, 25 7, 8, 16, 19 60.7% (17/28) 7.1% (2/28) 53.6% (15/28) 28.6% (8/28) 28.6% (8/28) 21.4% (6/28) 14.3% (4/28) A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 245 Table 9 ITSM process improvement initiatives. Initiative Primary studies Year Description TIPA (formerly called AIDA) 1, 2, 3, 4, 7, 13, 22, 23, 26, 27 2007 SPINI+ 4, 28 2007 NiCE(NOVE-IT) 6 2002 MITO 6, 25 2000 NOEMI 9, 10 2003 SPiCE Lite [ITSM] 11 2008 TickIT Plus 14 2008 SPICE 1-2-1 for ISO 20000 18 2007 IT service CMM 20 1998 ‘‘The TIPA model was inspired by ITIL best practices, with the goal to enable objective ITSM capability assessments. The references used to create the PRM and the PAM were the Service Support and Service Delivery books published by OGC.’’ (Primary study 1) ‘‘The SPINI-methodology was supplemented with process modeling methods and the process library was extended with IT Service Management processes (ITIL, ISO/IEC 20000).’’ (Primary study 28) ‘‘A process model for IT procurement, development, operation, and service provision (also called NOVE-IT) was developed.’’ ‘‘For the NOVE-IT project, it was decided to create an assessment model consisting of a process dimension adopted as-is from Part 2 of ISO/IEC TR 15504. The model is called NOVE-IT Capability dEtermination, or NiCE.’’ (Primary study 6) ‘‘A maturity model for the assessment of companies or parts of companies providing IT Operations services. The assessment is process based thus able to reveal the potentials for improvement. The model combines features of the SEI CMM and the assessment method of the EFQM.’’ (Primary study 25) ‘‘The development and experimentation of an IT process assessment methodology especially designed to be used in very small enterprises (VSEs).’’ ‘‘The processes themselves are based on a combined approach of ISO/IEC 15504 and the IT Infrastructure Library.’’ (Primary study 9) ‘‘SPiCE Lite [ITSM] supports the guided assessment of ITIL IT organization processes. SPiCE applies its own maturity level model to ITIL processes. It thus provides a qualitative evaluation of process maturity in accordance to the SPiCE-process maturity model (ISO/IEC 15504).’’ (Primary study 11) ‘‘ISO/IEC 15504, ISO/IEC 15288 and ISO/IEC 12207 will form the core around which TickIT Plus is designed: the capability model and the process structure. Each of other three standards: ISO/IEC 20000, ISO/IEC 27001 and ISO/IEC 25030, termed ‘Requirements Standards’, are all optional standards that can be included under the TickIT Plus certification.’’ (Primary study 14) ‘‘So ISO/IEC 15504 can be used as universal model for process assessment and process improvement. Following this idea I have defined an ‘ISO 20000 – PAM’, a process assessment model for IT Service Management according to ISO/ IEC 20000–1:2005. Based on the ISO 20000 – PAM we have implemented an Assessment Tool for IT Service Management: SPICE 1-2-1 for ISO 20000.’’ (Primary study 18) ‘‘We propose an Information Technology Service Capability Maturity Model (IT Service CMM) that can be used to assess the maturity of IT service processes and identify directions for improvement.’’ (Primary study 20) may be a reflection of a growing awareness of the importance of ITSM Process Improvement. Table 10 Classification of ITSM Process Improvement models. Assessment method Process reference model 5. Conclusions and future work ITIL ISO/IEC 20000 Self-developed SCAMPI – – MITO IT service CMM NiCE (NOVE-IT) ISO/IEC 15504-2 SPINI+ TIPA NOEMI SPICE lite [ITSM] SPINI+ TickIT plus SPICE 1-2-1 for ISO 20000 Fig. 2. Number of primary studies included in the review. 4.3. Temporal view of primary studies Fig. 2 shows the temporal distribution of primary studies resulting from the review carried out. Since most of the models were developed from 2006 onwards, there is a notable increment from this year in the number of published studies showing the experiences of their development and usage. This recent increase In this systematic review we have formulated a research question which aims to identify existing ITSM Process Improvement initiatives based on the ISO/IEC 15504 standard for process assessment and improvement. By examining the results obtained after the systematic review execution, we have detected nine different ITSM Process Improvement initiatives in which new models for the assessment of this kind of processes have already been developed. We have observed that all these models are composed of an ITSM process reference model and an assessment method. This review suggests that ISO/ IEC 20000, ITIL (V2 and V3) and CMMI-SVC are the ITSM process reference models most frequently used. Both SCAMPI and ISO/IEC 15504-2 have been used by these new ITSM Process Improvement models as process assessment methods. If we focus on the models based on ISO/IEC 15504, seven of the nine detected models use ISO/IEC 15504 conformant process assessment methods to assess the ITSM processes defined by ITIL or ISO/IEC 20000. Then, the results suggest that the feasibility of using the ISO/IEC 15504 measurement framework for ITSM process assessment and improvement has already been examined and proved during the last decade. With that same purpose, ISO started the development of an ITSM Process Reference Model (ISO/IEC 20000-4) and its Process Assessment Model (ISO/IEC 15504-8) conformant to the ISO/IEC 15504 assessment method. Bearing in mind the final goal of our research (the development of a new method with the necessary guidelines for the assessment according to ISO/IEC 15504 of both software lifecycle and ITSM processes reducing the amount of effort), further work would have to be performed. To meet this goal the first step would consist of examining SPINI+, TickIT Plus and SPICE 1-2-1 for ISO 20000, the three models which use the ISO/IEC 15504 assessment method to assess the ISO/IEC 20000 processes. These models will be 246 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 considered as a starting point for the assessment of the ISO/IEC 20000 processes. Then, our work will focus on the development of a new process assessment model based on the ISO/IEC 15504 measurement framework for the assessment and improvement of both the ISO/IEC 15504-5 software development processes and the ITSM processes provided by ISO/IEC 20000-4. As some processes of both process reference models are overlapped, the commonalities between them should be carefully analysed. Acknowledgments This work is supported by CICYT-TIN2010-20057-C03-03 ‘‘Simulación aplicada a la gestión de equipos, procesos y servicios’’, Sim4Gest. Appendix A. List of primary studies in the systematic review Next, selected primary studies in the development of this systematic review are presented. A.1. List of primary studies in the systematic review [1] B. Barafort, A. Rousseau, ‘‘Sustainable Service Innovation Model: A Standardized IT Service Management Process Assessment Framework’’, Software Process Improvement: EuroSPI 2009. Communications in Computer and Information Science, vol. 42, pp. 69–80, Springer-Verlag, Heidelberg, 2009. [2] B. Barafort, B. Di Renzo, O. Merlan, ‘‘Benefits resulting from the combined use of ISO/IEC 15504 with the Information Technology Infrastructure Library (ITIL)’’, International Conference on Product Focused Software Process Improvement (PROFES 2002). Lecture Notes in Computer Science, vol. 2559, pp. 314–325, Springer-Verlag, 2002. [3] B. Barafort, B. Di Renzo, V. Lejeune, S. Prime, J.-M. Simon, ‘‘ITIL Based Service Management measurement and ISO/IEC 15504 process assessment: a win–win opportunity’’, Proceedings of the 5th International SPICE Conference on Process Assessment and Improvement (SPICE 2005), Klagenfurt, Austria, 2005. [4] B. Barafort, D. Jezek, T. Mäkinen, S. Stolfa, T. Varkoi, I. Vondrak, ‘‘Modeling and Assessment in IT Service Process Improvement’’, Software Process Improvement: EuroSPI 2008. Communications in Computer and Information Science, vol. 16, pp. 117–128, Springer-Verlag, 2008. [5] B. Barafort, A. Renault, M. Picard, S. Cortina, ‘‘A transformation process for building PRMs and PAMs based on a collection of requirements - Example with ISO/IEC 20000’’, Proceedings of the International SPICE Conference on Process Improvement and Capability dEtermination (SPICE 2008), Nuremberg, Germany, 2008. [6] A. Cass, C. Völcker, P. Sutter, A. Dorling, H. Stienen, ‘‘SPiCE in Action - Experiences in Tailoring and Extension’’, Proceedings of the 28th Euromicro Conference (EUROMICRO 2002), Dortmund, Germany, 2002. [7] Cater-Steel, ‘‘Integration of service management with CMMIÒ and SPICE’’, 5th Annual SEPG Australia Conference, Gold Coast, Australia, August 2007. [8] Cater-Steel, ‘‘IT Service Departments Struggle to Adopt a Service-Oriented Philosophy’’, International Journal of Information Systems in the Service Sector, vol. 1, issue 2, pp. 69–77, 2009. [9] Di Renzo, C. Feltus, ‘‘Process assessment for use in very small enterprise: the NOEMI assessment methodology’’, Proceedings of the European Software Process Improvement Conference (EuroSPI 2003), Graz, Austria, December 2003. [10] Di Renzo, C. Feltus, S. Prime, ‘‘Collaborative management for ICT process improvement in SME: experience report’’, Proceedings of the European Software Process Improvement Conference (EuroSPI 2004), Trondheim, Norway, November 2004. [11] T. Goldschmidt, A. Dittrich, M. Malek, ‘‘Quantifying Criticality of Dependability-Related IT Organization Processes in CobiT’’, IEEE Proceedings of 15th Pacific Rim International Symposium on Dependable Computing (PRDC 2009), Shanghai, China, 2009. [12] Grandry, E. Dubois, M. Picard, A. Rifaut, ‘‘Managing the Alignment between Business and Software Services Requirements from a Capability Model Perspective’’, Towards a Service-Based Internet (ServiceWave 2008), Lecture Notes In Computer Science, vol. 5377, pp. 171–182, 2008. [13] R. Hilbert, A. Renault, ‘‘Assessing IT Service Management Processes with AIDA - Experience Feedback’’, Proceedings of the European Systems & Software Process Improvement and Innovation Conference (EuroSPI 2007), Potsdam, Germany, September 2007. [14] Irving, ‘‘TickIT Plus - the Future of TickIT!’’, TickIT International, issue 2Q08, pp. 3–7, 2008. [15] A. Kramer, ‘‘ISO/IEC 15504 and ITIL’’, International SPICE Days 2008, Prague, Czech Republic, June 2008. [16] D. Malzahn, ‘‘A service extension for SPICE?’’, Proceedings of the International SPICE Conference on Process Assessment and Improvement (SPICE 2007), Seoul, South Korea, May 2007. [17] D. Malzahn, ‘‘Assessing - learning - improving, an integrated approach for self assessment and process improvement systems’’, Proceedings of the fourth International Conference on Systems (ICONS 2009), pp. 126–130, Cancun, Mexico, March 2009. [18] A. Nehfort, ‘‘SPICE Assessments for IT Service Management according to ISO/IEC 20000–1’’, International SPICE Days 2007, Frankfurt, Germany, June 2007. [19] R. Nevalainen, M. Johansson, ‘‘Comparison of CMMI-SVC and ISO20000 – A Case Study’’, Proceedings of the European Systems & Software Process Improvement and Innovation Conference (EuroSPI 2008), Dublin, Ireland, September 2008. [20] Niessink, H. Van Vliet, ‘‘Towards Mature IT Services’’, Software Process - Improvement and Practice, vol. 4, issue 2, pp. 55–71, June 1998. [21] M. Picard, A. Renault, S. Cortina, ‘‘How to Improve Process Models for Better ISO/IEC 15504 Process Assessment’’, Systems, Software and Services Process Improvement (EuroSPI 2010), CCIS 99, pp. 130–141, 2010. [22] Public Research Centre Henri Tudor: B. Barafort, V. Betry, S. Cortina, M. Picard, M.St-Jean, A. Renault, O. Valdés, ‘‘ITSM Process Assessment Supporting ITILÒ’’, Van Haren Publishing, Zaltbommel, December 2009. [23] A. Renault, B. Barafort, ‘‘TIPA: 7 years experience with SPICE for IT Service Management’’, Proceedings of the European System & Software Process Improvement and Innovation Conference (EuroSPI 2011), Roskilde, Denmark, June 2011. [24] A. Rifaut, ‘‘Goal-Driven Requirements Engineering for Supporting the ISO 15504 Assessment Process’’, Software Process Improvement: EuroSPI 2005. Lecture Notes in Computer Science, vol. 3792, pp. 151–162, Springer-Verlag, 2005. [25] Q. Scheuing, K. Frühauf, W. Schwarz, ‘‘Maturity model for IT operations (MITO)’’, Proceeding of the 2nd World Congress on Software Quality, Yokohama, Japan, September 2000. [26] M. St-Jean, ‘‘TIPA to keep ITIL going and going’’, Proceedings of the European Systems & Software Process Improvement and Innovation Conference (EuroSPI 2009), Alcalá de Henares, Spain, September 2009. A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 [27] M. St-Jean, A.-L. Mention, ‘‘How to evaluate benefits of Tudor’s ITSM Process Assessment?’’, Proceedings of the International SPICE Conference on Process Improvement and Capability dEtermination (SPICE 2009), Turku, Finland, June 2009. [28] T. Varkoi, T. Makinen, ‘‘Proactive elicitation of software process improvements’’, Proceedings of the Portland International Conference on Management of Engineering & Technology (PICMET 2008), pp. 1576–1579, Cape Town, South Africa, July 2008. [10] [11] [12] [13] [14] [15] [16] References [17] Ò [1] Software Engineering Institute (SEI), CMMI-DEV, CMMI for Development (CMMI-DEV) Version 1.3, 2010. [2] International Organisation for Standardization (ISO/IEC), ISO/IEC 155041:2004 Information Technology – Process Assessment – Part 1: Concepts and Vocabulary, 2004. [3] A. Mas, E. Amengual, La mejora de los procesos de software en las pequeñas y medianas empresas (pyme). Un nuevo modelo y su aplicación en un caso real, Revista Española de Innovación, Calidad e Ingeniería del Software (REICIS) 1 (2) (2005) 7–29. [4] A. Mas, E. Amengual, A Method for the implementation of a quality management system in software SMEs, in: Proceedings of the Twelfth International Conference on Software Quality Management, British Computer Society, March 2004, pp. 61–74. [5] E. Amengual, A. Mas, Software process improvement in small companies: an experience, in: Proceedings of the European Systems & Software Process Improvement and Innovation Conference (EuroSPI 2007), Potsdam, Germany, September 2007. [6] A. Mas, B. Fluxà, E. Amengual, Lessons learned from an ISO/IEC 15504 SPI programme in a company, in: Proceedings of the European Systems & Software Process Improvement and Innovation Conference (EuroSPI 2009), Alcalá de Henares, Spain, September 2009. [7] A.L. Mesquida, A. Mas, E. Amengual, La madurez de los servicios TI, Revista Española de Innovación, Calidad e Ingeniería del Software (REICIS) 5 (2) (2009) 77–87. [8] International Organisation for Standardization (ISO/IEC), ISO/IEC 155042:2004 Software Engineering – Process Assessment – Part 2: Performing an Assessment, 2003. [9] International Organisation for Standardization (ISO/IEC), ISO/IEC NP TR 155048 Information Technology – Process Assessment – Part 8: An Exemplar Process [18] [19] [20] [21] [22] [23] [24] [25] 247 Assessment Model for IT Service Management, Standard Under Development, ISO/IEC JTC1/SC7 3798, NWI Proposal, 2007. Office of Government Commerce (OGC), ITIL Version 3 Service Strategy, 2007. Office of Government Commerce (OGC), ITIL Version 3 Service Design, 2007. Office of Government Commerce (OGC), ITIL Version 3 Service Transition, 2007. Office of Government Commerce (OGC), ITIL Version 3 Service Operation, 2007. Office of Government Commerce (OGC), ITIL Version 3 Continual Service Improvement, 2007. International Organisation for Standardization (ISO/IEC), ISO/IEC 200001:2005 Information Technology – Service Management – Part 1: Specification, 2005. International Organisation for Standardization (ISO/IEC), ISO/IEC 200002:2005 Information Technology – Service Management – Part 2: Code of practice, 2005. International Organisation for Standardization (ISO/IEC), ISO/IEC CD TR 200004 Information technology – Service Management – Process Reference Model, Standard Under Development, ISO/IEC JTC1/SC7 3797, NWI Proposal, 2007. Software Engineering Institute (SEI), CMMI-SVC, CMMIÒ for Services (CMMISVC) Version 1.3, 2010. B. Kitchenham, Guidelines for Performing Systematic Literature Reviews in Software Engineering Version 2.3, Technical Report EBSE-2007-01, Software Engineering Group, School of Computer Science and Mathematics, Keele University, and Department of Computer Science, University of Durham, July 2007. B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, J. Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering 28 (8) (2002) 721– 734. J. Biolchini, P. Gomes, A.C. Cruz, G. Horta, Systematic Review in Software Engineering, Systems Engineering and Computer Science Department, COPPE/ UFRJ, Technical Report RT-ES679/05, Rio de Janeiro, Brasil, May 2005. J.A. Calvo-Manzano, G. Cuevas, G. Gasca, T. San Feliu, State of the art for risk management in software acquisition, ACM SIGSOFT Software Engineering Notes 34 (4) (2009) 1–20. J.A. Calvo-Manzano, G. Cuevas, G. Gómez, J. Mejía, M. Muñoz, F. Rabbi, T. San Feliu, Service level management for IT services in small settings: a systematic review, in: Proceedings of the European Software Process Improvement Conference (EuroSPI 2009), Alcalá de Henares, Spain, September 2009. F.J. Pino, F. García, M. Piattini, Software process improvement in small and medium software enterprises: a systematic review, Software Quality Journal 16 (2) (2008) 237–261. D. Cruzes, M. Mendonça, V. Basili, F. Shull, M. Jino, Extracting Information from Experimental Software Engineering Papers, in: Proceedings of the XXVI International Conference of the Chilean Society of Computer Science (SCCC 2007), 2007, pp. 105–114.