Using a Partially Instantiated GQM to Measure the Quality of Mobile Applications Luis Corral, Alberto Sillitti, Giancarlo Succi Center for Applied Software Engineering Free University of Bozen-Bolzano Bozen-Bolzano, Italy Luis.Corral@stud-inf.unibz.it, {Alberto.Sillitti, Giancarlo.Succi}@unibz.it Abstract— Mobile application markets manage hundreds of thousands of products involving millions of downloads. The quality of these applications is normally controlled by market policies; however, there is no link between the quality goals of the mobile software market and the practices that have to be exercised to assure a compliant application. In this paper, we propose a GQM-based strategy to supply the mechanisms to measure the quality of mobile software products. By partially instantiating a GQM structure, we can have a schema previously furnished to consider beforehand the conditions of the mobile domain that typically impact the quality of the final product. Our approach offers to apply a strong quality model in a field that requires strong quality metrics to assess products generated in short development cycles, with a potential impact of millions of users, executed in a very limited environment. Keywords- Assessment, Evaluation, GQM, Metrics, Mobile, Product, Quality, Standards I. INTRODUCTION Mobile devices have evolved from being a communication tool toward becoming the primary end-user computing equipment. The introduction of smartphones and tablets promoted a massive growth in the creation of mobile applications, thanks to their operative organization and the capability of installing and removing applications at user space, opening a development opportunity previously bounded to telephone manufacturers and carrier companies. Handheld devices are able to carry out computer operations that cover a variety of purposes, from traditional voice communication and entertainment to business applications, mobile banking, and other complex operations. The offer and demand of mobile software applications spans in very large ranges: major mobile distribution channels manage hundreds of thousands of applications involving millions of downloads. This allows developers to distribute mobile “apps” in a wide scale, while users have access to a huge range of products. In order to maintain a standard quality level, application storefronts count on publishing guidelines that must be observed for an app to be considered for inclusion in market (mainly criteria related to adequacy of content, privacy, copyright protection, etc.). In this way, mobile apps are evaluated before being accepted for distribution and rated by other users after being downloaded. However, up to the day, mobile developers do not have a way to appraise the quality of their apps from an objective and quantitative point of view. Developers, for example, would like to know if an application meets  market  and  user’s  needs,  and  may  benefit  from  knowing   in a detailed fashion how a successful mobile application was implemented. In this paper, we propose a GQM-based strategy to supply the mechanisms to facilitate the definition of metrics to monitor the quality of mobile software products. By partially instantiating the definition of a GQM structure, we can have a schema that considers beforehand the conditions of the mobile domain that typically impact the quality of the final product. The rest of the paper is organized as follows: Section II reviews the related work; Section III presents our research questions; Section IV introduces our implementation strategy in the form of the Partially Instantiated GQM; Section V shows sample applications of our approach; Section VI outlines our future work and Section VII provides a summary and conclusions. II. RELATED WORK Under   the   premise   of   “high   quality   processes   deliver   high   quality   products”,   mobile   Software   Engineering   has produced an extensive body of knowledge aiming to determine what are the processes and practices that facilitate the creation of high quality, successful products. As the software product specializes in one domain, these processes may be customized and adapted to suit the specific needs of the environment [1, 2]. Moreover, software metrics should be designed according to the needs of the business and the possibilities of the particular ecosystem [3, 4, 5]. When speaking about mobile software, the diverse and heterogeneous factors in which these applications rely (e.g., autonomy, connectivity, resource limitation, etc.) build up an environment that is complex and fault prone [6]. In consequence, mobile software developers should consider customized development practices that aid to ensure the quality of their applications when performing in such a limited and particular environment. To overcome this need, there are research works that aim to furnish comprehensive frameworks to assure and evaluate the quality of mobile software product, both from process and product standpoints. From the software process point of view, different development models have been proposed to accommodate specific needs of the mobile environment throughout activities, tasks and reviews conducted during the development process. A number of these software development methodologies concur on the fact that Agile-inspired practices are the most appropriate for conducting mobile software projects [7]. The suitability of Agile for the fulfillment of objectives mobile software development for mobile devices has been discussed in detail [8, 9], and Agile practices (Pair Programming, Refactoring, Test Driven Development) that have been analyzed have shown benefits in large industrial and experimental settings [10-15]. Some of these practices were adapted and incorporated to the mobile software domain. Building on top of this solid approach, we can point out Mobile-D [16], Hybrid Development [17], MASAM [18], Scrum [19] and Scrum/Lean Six Sigma [20]. Although the proposed methodologies show a thorough effort on process assurance, they miss to consider the target environment, disregarding environment-specific conditions that might enrich their quality and measurement activities. Focusing on product quality, research works are less extensive. Current proposals focus on methods to assess the quality of mobile applications [21]. Dantas et al. [22] presented a review of testing requirements particular to applications developed for mobile devices that can be easily adapted, incorporating detailed activities of product measurement. Spriestersbach and Springer [23] identified relevant challenges in the development of mobile web application and related such challenges into the quality characteristics described by the ISO/IEC 9126 quality standard. Mantoro [24] suggests adjustments to the same ISO quality model for assuring the quality of context-aware applications. These perspectives are highly relevant to the matter at hand, as many conditions found on the mobile web and context-aware applications also hold on general purpose mobile applications. By analyzing the process and product methodologies already present in literature, we note that while we have a variety of works concerning mobile software quality, we still miss the link between the quality goals of the mobile software and the processes that have to be exercised to develop applications that fulfill such expectations. In addition, there is a lack of works regarding how to measure the level of accomplishment of a mobile application by using quality attributes and domain-specific software metrics. III. RESEARCH PROBLEM To be able to evaluate or predict the quality of a mobile software product, it is necessary to count on reference points that permit to compare the product against a series of or expected characteristics present on successful applications, in such way that a given product can be evaluated objectively and quantitatively. Regardless of the platform or operating system used (e.g., Android, iOS, etc.), mobile devices have to cope with several physical and infrastructure limitations (memory, CPU, battery, networks, etc). Furthermore, a mobile application typically encompasses heterogeneous technologies that have to interact appropriately to provide a satisfactory experience. As a consequence, traditional Software Engineering practices should bear additional considerations. Understanding the context of the mobile domain and its limitations is of vital importance to guarantee the success of a mobile software project; a solid knowledge on these aspects enables developers to deliver better applications, and allows users to select and purchase them in a more conscientious way. Several mature and recognized quality standards have been made available: ISO/IEC 9126, ISO/IEC 25010, McCall, and others; however, their effectiveness and applicability on the mobile software domain can be challenged as they do not take into account some of the conditions of mobile software that are not present in traditional desktop software products. Thus, it is necessary to consider the customization of these mature frameworks to fit the particular needs of the domain. For instance, there are development and appraisal models for Open Source Software (OSS), (e.g., QualiPSo OMM [25], MOST [26] or QSOS [27]). These models pick up from other wellgrounded software quality models like CMMi, and introduce customizations to provide the necessary procedures to evaluate Open Source Software. The strategy followed by these models is establishing the boundaries of the OSS scope, pointing out and emphasizing domain-specific needs, and tailoring the quality model to enhance it in terms of applicability and effectiveness. Since we do not count on a quality model that defines goals, processes, policies and metrics that specifically fit the conditions existing in the mobile execution ecosystem, we can follow the example set by OSS to build a mobilespecific quality framework on top of mature software quality models, bearing in mind the scope, environment-specific concerns and, and platform-specific quality needs. Given the high impact of the mobile software applications, it is of paramount importance to be able to analyze them to provide a sustained judgment, useful for the development, acquisition, recommendation, or deployment of mobile software products. To achieve this, we need to learn from traditional and innovative quality frameworks that have proven their value on other environments, (e.g., desktop, business, embedded) and modify them incorporating domain-specific requirements (e.g., usability matters, market awareness, etc.), and target-specific needs (e.g., physical and infrastructure limitations) To appraise the quality of the mobile software product, it is compulsory to have a clear understanding of the attributes that drive it, and to have solid foundations about the expectations of the stakeholders. With the purpose of finding these solutions, we formulated a research question: RQ: How mobile software products can be analyzed to provide a solid and objective notion of its quality? This means that we set as the main objective of this work to achieve the capability of associating the critical requirements and success factors of a mobile product with quality characteristics that can be measured and controlled. To provide an answer, we need first to survey what is typically expected from a mobile application from the viewpoint of the involved stakeholders. Many of the expectations of traditional software hold in the mobile scope; however, it is essential to deepen this analysis into the mobile domain. After this, it is necessary to analyze what are the conditions present in mobile ecosystem, not present in other environments, which have a significant impact on the quality of the software product. At this point, we will be in a position to set and find the answer of an important sub-question: of the final product. By applying the partially instantiated GQM, the time to produce a new metric can be significantly shortened and it will be ensured that such metrics will be beneficial to keep track of mobile-specific quality drivers. How can we measure the quality of a mobile software product? The answer of this sub-question requires the definition of a collection of metrics that help to approximate quantitatively the quality of a mobile product bearing in mind attributes of the product that are highly relevant for users, developers, execution environments and application markets. Creating partial instantiation of a GQM allows us to establish a core, extensible model that can be tailored, depending on the focus of the analysis. The goal will always be the analysis of the quality of the product: IV. IMPLEMENTATION APPROACH Our research question requires conducting a comprehensive survey to select the most important quality requirements set upon apps. Mobile application markets already count on policies that can be considered the minimum quality expectation for a mobile app. These publication guidelines are highly influential (i.e., applications not compliant with them are not included on the market, or discontinued). We may start from those rules to draw an outline of what a mobile application should look for, what constraints should it bear, and how should it perform. Therefore we may consider enhancing them with additional requirements surveyed from other sources (e.g., mobile software development experiences, processes etc). Then, the answer to our research sub-question requires an accurate and comprehensive definition of domain-specific metrics. To this end, we propose to use the Goal-QuestionMetric (GQM) approach. The GQM approach proposes to define a goal, refine such goal into questions, and introduce metrics that collect the necessary information to answer the questions. A goal represents the conceptual level: it is defined for an object for a variety of reasons, with respect to various models of quality, from various points of view and relative to a particular environment. Questions represent the operational level: A set of questions try to characterize the object of measurement with respect to a selected quality issue, and to determine its quality from a selected viewpoint. Finally, metrics represent the quantitative level: data is associated with every question to answer it in a quantitative way [28, 29]. To evaluate the quality of a product in the mobile domain, it is necessary to relate the mobile-specific characteristics with measurable attributes. For instance, from an operative point of view, we should be aware that mobile software runs on a slow processor, with limited input means, powered by a battery, dealing with high autonomy requirements. Among others, these constraints pose common requisites   on   the   application’s   quality, and such constraints may be characterized to understand how the mobile environment, by itself, introduces new quality requirements that can be also analyzed and traced. To maximize the profit of our approach, we propose to summarize the common requisites and prepare a GQM baseline that can be partially instantiated. This GQM should work as a template that can be detailed and customized depending on the scope of the desired quality assessment. We believe that a GQM can be preliminarily furnished to facilitate an efficient approximation of the quality from a viewpoint relevant to the mobile ecosystem, considering beforehand the conditions that typically arise on the mobile domain and that impact the quality G.1: “Analyze the mobile software product for the purpose of evaluating it with respect to the quality, from the view point of developers and customers, in the context of execution environment and application markets”. The questions, on the other hand, can be preliminarily set to survey the characteristics of the application and to evaluate it with respect to an outstanding attribute (X) that relates directly to the issue that the GQM model attempts to solve: Q.1: “What   is   the   current   performance   of   the   application   with  respect  to  attribute  X?” Q.2: “Is  the  current  performance  of  attribute  X  satisfactory   from  the  viewpoint  of  the  developer  and  the  user?” Q.3: “Is  the  current  performance  of  attribute  X  acceptable   from  the  viewpoint  of  the  application  market?” Finally, it is not possible to supply beforehand a fullyequipped set of metrics, since they shall be proposed depending on each case. Then, it should be provided a data source that leads to answer the questions from an unbiased and quantitative point of view. V. SAMPLE APPLICATION Let us consider two sample applications of our approach, to evidence its actual feasibility. In our first example, we want to evaluate the quality of the Facebook Android App in terms of its suitability with the battery life. Based on our partial instantiation of the GQM, we translate it into the following interpretation: G.1: “Monitor  the  quality  of  the  product  for  the  purpose  of   measuring its energy consumption from the point of view of the user in the context of the mobile execution environment”. Different questions need to be set, including: Q.1: “What   is   the   current   performance   of   the   application   with  respect  to  energy  consumption?” Q.2: “Is   the   current   performance   of   energy   consumption   satisfactory  from  the  viewpoint  of  the  user?”   Q.3: “Is   the   current   performance   of   energy   consumption   acceptable  from  the  viewpoint  of  the  application  market?”   To solve these questions, a number of metrics can be calculated, for example: M.1: Number of milliwatts consumed by the mobile phone while the application is executed. M.2: Percentage of battery drained while the application is executed. Such values will supply an accurate notion of the amount of energy spent by the application. To determine the answers to the questions and the accomplishment of the goal, the values obtained from the metrics should be evaluated with respect to a performance standard and the market policy concerning to energy consumption and battery drains. In the second application, we want to evaluate the quality of the Trip Advisor Android App in terms of its observance of user’s   privacy   while   retrieving   his   or   her   physical   position.   From the partial instantiation of the GQM, we obtain: G.1 “Monitor  the  quality  of  the  product  for  the  purpose  of   assessing its compliance with the privacy policy from the point of view of the user in the context of the mobile execution  environment”. Possible questions are: Q.1: “What   is   the   current   performance   of   the   application   with  respect  to  privacy  compliance?”   Q.2: “Is  the  current  performance  of  the privacy compliance satisfactory  from  the  viewpoint  of  the  user?”   Q.3: “Is  the  current  performance  of  the  privacy  compliance   acceptable  from  the  viewpoint  of  the  application  market?”   We can define the following metrics: M.1: Amount of time spent on accesses to GPS data while the application is working. M.2: Amount of time spent on network access while the application is working. In this way, one can analyze if the app is retrieving the user’s  position  only  under  his  or  her  consent.  To  complete  the   evaluation of the questions and the achievement of the goal, the values obtained from the metrics should be evaluated with respect to a performance standard, and the market policy concerning to privacy management. Since the metrics may be hard to determine, we foresee the selection of a mature software assurance framework that can be tailored to the needs given by the mobile ecosystem and the mobile product. Our intention is to build on top of the ISO/IEC 25010:2011 quality standard [30], and analyze its quality characteristics and attributes, focusing in those that are applicable to the quality of the mobile software product context. ISO/IEC 25010:2011 defines quality characteristics grouped   in   two   orthogonal   dimensions:   A   “product   quality   model”  that  relates  to static properties of software and dynamic properties  of  the  computer  system,  and  a  “quality  in  use  model”   that relates to the outcome of interaction when a product is used in a particular context of use, making it very suitable to the viewpoint of our GQM. VI. FUTURE WORK There is still an extensive effort to carry out in order to accomplish the goals of the underlying work. With this proposal and further experimentation, we aim to contribute to state of the art on mobile Software Engineering by enabling mechanisms to assure and monitor the quality of the mobile software product, based on real-world quality expectations and real-world market awareness, delivering a methodology to identify metrics that enable substantiated quantitative analysis. We also envision to provide the means to assist in decision making to establish strategies or recommend practices to optimize the development processes in mobile software projects. This work also aims to share other relevant questions that have been marginally established at this stage, such as: (a) What is the extent in which the execution environment affects the quality of a mobile application? (b) Will the quality metrics of a given mobile app relate to the restrictions and constraints imposed by the application market (i.e., the most restrictive the app market is, the less defects should be found in an application downloaded from it?) In a long-term scope, establishing a strategy for software quality assessment for mobile applications will be helpful for researchers, developers and users to: (i) Determine the capacity of the mobile software application to meet specific requirements and demands. (ii) Identify quality requirements addressable when designing and implementing the mobile software product, and implement metrics that allow to measure quality attributes and track quality characteristics. To verify the accuracy and usefulness of this approach, it is necessary to conduct a piloted multi-dimensional evaluation of diverse applications using the partially instantiated GQM and its associated quality attributes. We will design and deploy a case study that will supply the empirical data required to analyze the effectiveness this approach a real industrial setting. The analysis will be executed by designing and implementing product metrics to analyze the applications developed by a specialized mobile software company. The definition of the metrics will be done after a partially instantiated GQM created in collaboration with project managers, developers, testers and pilot users. Metrics will be co-related to quality attributes present in ISO/IEC 25010:2011, to guarantee the implementation of both the product quality and quality in use models. Finally this study will be complemented with developer’s   data collected non-invasively, to consider the impact of different levels of experience and seniority [31-35]. As means of validation, we propose that the product quality metrics obtained after the analysis shall be compared to the number  of  downloads  and  to  the  user’s   rating obtained by the product in the application store, to identify if there is correlation between the metric-oriented product assessment and the user-provided product assessment. VII. CONCLUSIONS The expansion of mobile platforms as high-end, ubiquitous computing equipment requires to understand the mobile environment to create the best strategies to assure the quality of the mobile software product from a domain-specific point of view. Our approach pursues to apply a specific, partially instantiated GQM that helps to identify goals, set questions and design effective metrics considering in advance the conditions that exist on the mobile domain and that impact significantly the quality of the final product. Mobile applications stand out from other traditional software products due to a highly competitive market and a potential impact on millions of users. The need of assuring the development of high-quality mobile products becomes an imperative, and demands extensive investigation and experimentation from the academic and practitioner point of view. The answers to our research question will enable us to relate the real-world expectations on the mobile software products with measurable characteristics, so that developers can deploy quality strategies for their mobile software projects, and final customers can conduct trustable evaluation efforts before purchasing or deploying a new mobile application. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] Maurer, F., Succi, G., Holz, H., Kötting, B., Goldmann, S., Dellen, B.; Software process support over the Internet. In Proceedings of the 21st International Conference on Software Engineering, pp. 642-645. 1999. Benedicenti, L., Succi, G., Vernazza, T., Valerio, A.; Object oriented process modeling with fuzzy logic. In Proceedings of the 1998 ACM Symposium on Applied Computing, pp. 267-271. 1998. Pedrycz, W., Succi, G., Chun, M.G.; Association analysis of software measures. International Journal of Software Engineering and Knowledge Engineering, 12 (3) , pp. 291-316. 2002. Scotto, M., Sillitti, A., Succi, G., Vernazza, T.; A relational approach to software metrics. In Proceedings of the ACM Symposium on Applied Computing, pp. 1536-15408. 2004. Pedrycz, W., Succi, G., Musïlek, P., Bai, X.; Using self-organizing maps to analyze object-oriented software measures. Journal of Systems and Software, 59 (1) , pp. 65-82. 2001. Corral L., Georgiev A. B., Sillitti A., Succi G.; A Method for Characterizing Energy Consumption in Android Smartphones. 2nd International Workshop on Green and Sustainable Software (GREENS 2013) in connection with ICSE 2013, pp. 38-45. 2013. Corral L., Sillitti A., Succi G.; Software Development Processes for Mobile Systems: Is Agile Really Taking Over the Business?. 1st International Workshop on Mobile-Enabled Systems (MOBS 2013) in connection with ICSE 2013, pp. 19-24. IEEE. 2013. Abrahamsson, P., Warsta, J., Siponen, M., Ronkainen, J.; New directions on Agile methods: A comparative analysis. In Proceedings of the International Conference on Software Engineering. 2003. Abrahamsson, P.; Mobile software development: the business opportunity of today. In Proceedings of the International Conference on Software Development, pp. 20-23. 2005. Russo B., Scotto M., Sillitti A., Succi G.; Agile Technologies in Open Source Development. IGI Global, USA, ISBN 978-1-59904-681-5. 2009. Di Bella E., Fronza I., Phaphoom N., Sillitti A., Succi G., Vlasenko J.; Pair Programming and Software Defects – a large, industrial case study. Transactions on Software Engineering, IEEE. To appear (DOI: 10.1109/TSE.2012.68). (n.d.) Sillitti, A., Succi, G., Vlasenko, J.; Understanding the impact of pair programming  on  developers’  attention: a case study on a large industrial experimentation. In Proceedings of the International Conference on Software Engineering. pp. 1094-1101. 2012. Moser R., Sillitti A., Abrahamsson P., Succi G.; Does refactoring improve reusability?. 9th International Conference on Software Reuse (ICSR-9). 2006. Moser R., Abrahamsson P., Pedrycz W., Sillitti A., Succi G.; A case study on the impact of refactoring on quality and productivity in an agile team. In Proceedings of 2nd IFIP CEE-SET. 2007. Janes A., Remencius T., Sillitti A., Succi G.; Managing Changes in Requirements: an Empirical Investigation. Journal of Software: Evolution and Process, Wiley, to appear. (n.d.) Abrahamsson, P., Hanhineva, A., Hulkko, H., Ihme, T., Jäälinoja, J., Korkala, M., Koskela, J., Kyllönen, P., Salo, O.; Mobile-D: An Agile [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] approach for mobile application development. In Proceedings of OOPSLA’04. 2004. Rahimian, V. Ramsin, R.; Designing an Agile methodology for mobile software development: A Hybrid Method engineering approach. In Proceedings of the 2nd International Conference on Research Challenges in Information Science. 2008. Jeong, Y.J., Lee, J.H., Shin, G.S.; Development Process of Mobile Application Software Based on Agile Methodology. Proceedings of the 10th International Conference on Advanced Communication Technology, pp. 362-366. 2008. Scharff, C., Verma, R.; Scrum to support mobile application development projects in a just-in-time learning context. Proceedings of the 2010 Workshop on Cooperative and Human Aspects of Software Engineering (CHASE 2010) in connection with ICSE, pp. 25-31. 2010. Cunha, T.F.V., Dantas, V.L.L., Andrade, R.M.C.; SLeSS: A Scrum and Lean Six Sigma integration approach for the development of software customization for mobile phones. In Proceedings of the 25th Brazilian Symposium on Software Engineering, pp. 283-292. 2011. Ryan, C., Rossi, P.; Software, performance and resource utilisation metrics for context-aware mobile applications. In Proceedings of the 11th IEEE International Software Metrics Symposium. 2005. Dantas, V. L. L., Marinho, F. G., da Costa, A. L., and Andrade, R. M. C.; Testing requirements for mobile applications. In Proceedings of the 24th International Symposium on Comp. and Information Sciences, pp. 555-560. 2009. Spriestersbach, A. and Springer, T.; Quality attributes in mobile web application development. In Proceedings of the 5th International Conference Product Focused Software Process Improvement, LNCS vol. 3009, pp. 120-130. 2004. Mantoro, T.; Metrics Evaluation for Context-Aware Computing. In Proceedings of the 7th International Conference on Advances in Mobile Computing & Multimedia, 2009. Petrinja, E., Nambakam, R., Sillitti, A.; Introducing the Open Maturity Model. In Proceedings of the 2nd Emerging Trends in FLOSS Research and Development Workshop in connection with ICSE. 2009. Del Bianco, V., Lavazza, L., Morasca, S., Taibi, D.; Quality of Open Source Software: The QualiPSo Trustworthiness Model. In Proceedings of the 5th International Conference on Open Source Systems. 2009. Petrinja E., Sillitti A., Succi G.; Comparing OpenBRR, QSOS, and OMM Assessment Models. In Proceedings of the 6th International Conference on Open Source Systems. 2010. Basili, V, Caldiera, G., Rombach, D.; The Goal Question Metric Approach. Chapter in Encyclopedia of Software Engineering. Wiley, ISBN: 1-54004-8. 1994. Basili, V.; Applying the goal question metric paradigm in the experience factory. In Proceedings of the 10th Annual Conference of Software Metrics and Quality Assurance in Industry. 1993. International Organization for Standardization. ISO/IEC 25010:2011 Systems and software engineering. Quality Requirements and Evaluation System and software quality models, www.iso.org. 2011. Scotto, M., Sillitti, A., Succi, G., Vernazza, T.; A non-invasive approach to product metrics collection Journal of Systems Architecture, Vol. 52 (11) pp. 668-675. 2006. Corral, L., Sillitti, A., Succi, G., Strumpflohner, J., Vlasenko, J.; DroidSense: a mobile tool to analyze software development processes by measuring team proximity. Proceedings of TOOLS 2012, pp. 17-33. LCNS vol. 7304. 2012. Scotto, M., Sillitti, A., Succi, G., Vernazza, T.; A non-invasive approach to product metrics collection Journal of Systems Architecture, Vol. 52 (11) pp. 668-675. 2006. Sillitti, A., Succi, G., Vlasenko, J.; Toward a better understanding of tool usage: NIER track. In Proceedings of the International Conference on Software Engineering, pp. 832-835. 2011. Fronza, I., Sillitti, A., Succi, G., Vlasenko, J.; Understanding how novices are integrated in a team analysing their tool usage. Proceedings of the 2011 International Conference on Software and Systems Process, pp. 204-207. 2011.