A review of testing cloud security

374 Int. J. Internet Technology and Secured Transactions, Vol. 8, No. 3, 2018 A review of testing cloud security Eric Zenker and Maryam Shahpasand* Asia Pacific University of Technology and Innovation (APU), Technology Park Malaysia, Bukit Jalil, Kuala Lumpur 57000, Malaysia Email: z.eric@posteo.net Email: Maryam.shahpasand@apu.edu.my Email: M.shahpasand@gmail.com *Corresponding author Abstract: The cloud computing adoption process is constantly advancing whereas the security of clouds is still one of the major concerns of clients to adopt and use the new computing paradigm. To ensure a high level of security of cloud services and applications, testing is an appropriate approach to detect possible vulnerabilities before real case scenarios occur. Thus, many academic papers have been published to identify and address challenges in cloud security. However, most of the researchers focused on TaaS rather than on testing the cloud, which led to a current gap in academics. This paper presents a systematic literature review of testing cloud security. The authors elucidate a general and consistent topic overview, beginning with defining and introducing key terms. Furthermore, gaps in recent related publications are revealed, hence prospective research implications are pointed out to foster the understanding and relations of current research fields. Keywords: cloud computing; software as a service; SaaS; security; testing as a service; threat; vulnerability. Reference to this paper should be made as follows: Zenker, E. and Shahpasand, M. (2018) ‘A review of testing cloud security’, Int. J. Internet Technology and Secured Transactions, Vol. 8, No. 3, pp.374–397. Biographical notes: Eric Zenker graduated with a Bachelor’s in Industrial Engineering and Business Administration specialised in Computer Science from the University of Applied Sciences Merseburg, Germany in 2014. Subsequently, he was employed as a Business Application Consultant for SAP privilege and test management. Nowadays, he studies his Master’s in IT Management at the Staffordshire University on the campus of the Asia Pacific University of Technology and Innovation (APU) in Kuala Lumpur, Malaysia. His research interests include security-related IT topics with a major focus on eDiscovery and digital forensic. Maryam Shahpasand is an Academic Staff in the Asia Pacific University of Technology and Innovation (APU) and the Chair of FSec (Forensic and Cyber Security) Research Centre. She obtained her Bachelor and Master’s in Software Engineering and completed her PhD in Security in Computing at the University Putra Malaysia (UPM). She received the CIEH, CIHFI, CISSP Skills and CCME certificates and won gold, silver and bronze medals for security and forensic products. She is a member of IEEE Computer Society and her areas of specialisation are computer and network security, smart phone forensic and digital investigation. Copyright © 2018 Inderscience Enterprises Ltd. A review of testing cloud security 1 375 Introduction Today, cloud computing is a trendy and state-of-the-art solution in the information technology (IT) sector. Especially, organisations benefit from particular advantages of cloud computing like increased scalability and portability resulting in enhanced efficiency and cost reduction (Singh et al., 2016). Figure 1 shows that the global revenue for software as a service (SaaS) increased by 14.8% in 2016, which by implication means that the adoption process constantly advances. Evermore businesses recognise the benefits and shift their processes to the cloud (Statista Inc., 2016). Figure 1 Worldwide SaaS revenue (see online version for colours) Source: Statista Inc. (2016) Figure 2 Cloud challenges 2015 vs. 2016 (see online version for colours) Source: RightScale Inc. (2016) 376 E. Zenker and M. Shahpasand However, the adoption of cloud computing requires the awareness of various influencing factors. Privacy, trust and security still remain challenges for cloud adopters (Chang et al., 2016). RightScale Inc. (2016) conducted a survey on current cloud computing challenges (Figure 2). The sample rated security concerns second-highest in 2016 whereas it dropped from the top rank in 2015 by just a slight increase of 1% the following year. Cloud computing often handles sensitive data and clients of cloud solutions are partially confronted with a loss of authority of their own data due to a responsibility shift to the cloud service provider (CSP) (Ali et al., 2015). Thus, such systems are likely targets for security attacks causing radical on costs such as data modification or downtimes. Among software security incidents are exploited vulnerabilities a high portion. To identify these vulnerabilities and to ensure application security, security testing techniques are important and effective measures for improvement (Felderer et al., 2016). After the new cloud computing trend emerged in 2008/2009, most research focused on the definition and understanding of the term as well as on the identification of challenges and benefits (Buyya et al., 2008; Armbrust et al., 2010; Riungu-Kalliosaari et al., 2016). Following, there has been a consensus in academics, that security is among the main risks and challenges in adopting and using cloud solutions (Ali et al., 2015; Singh et al., 2016). For instance, Tao et al. (2012) developed a universal encryption framework for full data confidentially. 1.1 Problem statement In the early state of the newly emerged cloud computing paradigm, most researchers focused on a broader and coherent understanding, including definitions, challenges and benefits (Armbrust et al., 2010; Riungu-Kalliosaari et al., 2016). Subsequently, security of cloud environments became one of the most crucial concerns in adopting and using the new technology (Ali et al., 2015; Singh et al., 2016). The recent survey of RightScale Inc. (2016) revealed, that security challenges are the second highest concerns in cloud computing. Distributed systems are possible targets for attacks causing radical extra charges such as data modifications or downtimes. Data loss or leakage represents 24.6% and cloud-related malware 3.4% of threats causing cloud outages (Ko and Lee, 2013). Most of the software security incidents are exploited vulnerabilities. Hence, Akhgar (2016) recommends developing security metrics to identify vulnerabilities. To ensure application security, security testing techniques are important and effective countermeasures for improvement (Felderer et al., 2016). Thus, implemented systems should be tested by the use of analytical techniques and engineering principles to detect security issues as early as possible (Bos et al., 2014). However, according to Shrivastva et al. (2014), is security testing one of the major challenges in cloud testing environments. Besides, Nachiyappan and Justus (2015) indicated that present cloud security testing has many open queries, such as quality assurance and security validation. The authors also stated the challenge of testing security measures in cloud environments. Kumar and Singh (2014) revealed the research issue of performing quality checks within cloud environments. Beyond, Madan et al. (2016) pointed out the need to develop an approach for cloud privacy testing. Although the body of knowledge on cloud testing is growing, the literature review reveals an enormous gap of sophisticated security testing approaches for testing the cloud. Researchers mostly focused on test as a service (TaaS) rather than on testing the cloud. A review of testing cloud security 377 1.2 Significance of this study The recent survey of RightScale Inc. (2016) revealed, that security challenges are the second highest concerns in cloud computing. Moreover, according to Shrivastva et al. (2014), is security testing one of the major challenges in cloud testing environments. Thus, it is essential to investigate recent publications in the field. The main driver for security testing is the medium through which clouds are accessed. Figure 3 shows threat categories that caused cloud outages whereby data loss or leakage represents 24.6% and cloud-related malware 3.4%. Moreover, Christophe et al. (2014) revealed that from 2007 to 2013 cloud services had been down in a total number of 2,595.75 hours, resulting in a conservative estimate of $480,647,930 costs. Hence, it is important to reveal weak spots and to identify threats. One of the reasons stated the Dutch national cyber security research agenda that there is a lack of secure system designs. Thus, implemented systems should be tested by the use of analytical techniques and engineering principles to detect security issues as early as possible (Bos et al., 2014). However, Nachiyappan and Justus (2015) indicated that present SaaS security testing has many open queries, such as quality assurance and security validation. Gao et al. (2013b) stated that in SaaS environments, security testing is particularly important in terms of multi-tenancy. Therefore, the author aims to design a conceptual model for cloud security testing. Furthermore, Akhgar (2016) said security-by-design is an essential part of service quality. Hence, Akhgar (2016) recommends developing security metrics to identify vulnerabilities. This can be done by building open test beds. As a result, the scholar develops an experimental testbed to evaluate the efficiency of the proposed model by using security measures. Figure 3 Cloud outages by threat category (see online version for colours) Source: Ko and Lee (2013) 378 E. Zenker and M. Shahpasand In conclusion, security became one of the most important requirements within clouds and is a mandatory measure in SaaS environments (Nachiyappan and Justus, 2015). As so, cloud users need to assure the level of security of their used cloud services, which can preferably be done by testing the security of these applications. This survey will facilitate the understanding of the security of cloud services and furthermore, point out the vulnerabilities and threats as well as how the level of security can be ensured. 2 Cloud computing Cloud computing is “[…] a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources […] that can be rapidly provisioned and released with minimal management effort or service provider interaction” (Mell and Grance, 2011). According to Ruan et al. (2013) defines Gartner the cloud paradigm as a computing style where elastic and scalable IT resources are provisioned, like a service to various external clients via the use of the internet. Alam et al. (2015) defined it a parallel as well as distributed scheme comprising pooled virtualised and interconnected resources. Based on, between vendor and client negotiated, service level agreements (SLAs), are those capabilities dynamically presented and provisioned. According to Oliveira et al. (2015), refers the cloud paradigm to services and utilisations, which are accessed via the internet and available through a network. Besides, there are many actors involved in cloud environments. Figure 4 illustrates an overview of the entities. A cloud consumer is an organisation or individual that uses cloud services. A CSP is the purveyor of such services and products. In between these two parties acts the cloud broker to liaise through the complexity of various cloud computing offers. The cloud auditor evaluates and reports independently the level of security and performance of cloud services to governments. Lastly, the cloud carrier is responsible for transferring the data, akin to electric grid power distributors (Hogan et al., 2013). Figure 4 Cloud actors (see online version for colours) Source: Hogan et al. (2013) 379 A review of testing cloud security 2.1 Characteristics As shown in Table 1, the pay-as-you-go paradigm consists of five main characteristics, by to the US National Institute of Standards and Technology (NIST) (Mell and Grance, 2011). Cloud characteristics Table 1 Characteristics Description On-demand self-service Computing capabilities are, sans human interaction, automatically provisioned and consumed as need. Broad network access Capabilities are accessible via standardised network mechanism by heterogeneous client platforms. Resource pooling Within a multi-tenant model are technical capabilities virtually and physically pooled to dynamically service various client demands Rapid elasticity To serve scaled consumer demands, technical resources can be flexible offered and released. Measured service Capabilities are automatically controlled by metering measurements to tweak resource utilisation. Multi-tenancy This non-essential characteristic had been added by the cloud security alliance (CSA) and compromises the multiple and inter-organisational client use of a single capability. Auditability and certifiability Referred to multi-tenancy, measures enable affiliates to check the degree of service compliance. Source: Mell and Grance (2011), Jula et al. (2014), Ali et al. (2015) 2.2 Service models Cloud services are on demand available from the CSP’s servers, other than provisioned from the organisation’s own on-premises servers, to clients through the internet. Depending on the service models, the cloud vendor offers servers, hardware, storage, network components, computing platforms or applications for on-demand use. The customer usually pays per use. The CSP possesses the infrastructure and is responsible for hosting and maintaining. The most common examples are Google and Amazon among others (Iyer, 2016). Cloud services are beyond classified within several service models illustrated in Table 2. In chronological order, increases the severity of adoption. Moreover, these approaches are not specific enough regarding market requirements. Therefore, anything as a service (AaaS) is used to denote it in general. To name but a few storage as a service, big data as a service, database as a service, security as a service, etc. (Zafar et al., 2017). 380 Table 2 E. Zenker and M. Shahpasand Cloud service models Service model Description Software as a service (SaaS) A client utilises online-provisioned application software sans any authority to control underlying capabilities, operating systems or infrastructure. Platform as a service (PaaS) A supplier allocates as infrastructure, including technical capabilities sans dedicating control over the operating system or infrastructure to consumers to deploy customised or own application. Infrastructure as a service (IaaS) This approach only provides a plain computing infrastructure as well as technical resources for clients to virtually run their own operating system and applications Anything as a service (AaaS) It is a collective and an interchangeable model in respect of specific resources. Source: Mell and Grance (2011), Singh et al. (2016) 2.3 Deployment models Table 3 exemplifies the infrastructure of cloud services within deployment models. The private cloud can be deployed within an organisation and accessed over a local network like the intranet. A simple example is a file sharing system within a business unit. In general, the private cloud features the exclusive use by an individual organisation utilised by multiple clients. In addition, the organisation is responsible for all system matters which leverages data security as rules and policies can be integrated into the corporate standards (Mell and Grance, 2011; Jula et al., 2014). The public cloud, on the other hand, is only and inevitable provisioned over the internet because it is designed for open general public use. On one side, the provider offers various applications for lower costs, but on the other side, he defines policies and rules. More in detail, this solution can be deployed with or without internal IT involvement. However, both approaches imply changes in internal processes as well as in roles and responsibilities. The disadvantages are that the client never knows where his data is stored and therefore security standards and legislations are vague and partially unknown (Majendran, 2013; Jula et al., 2014). Table 3 Cloud deployment models Deployment model Description Private cloud Services are not public offered to multiple clients within a single organisation by an on or off premises data centre. Community cloud This model extends private clouds while providing access to a particular community with mutual interests. Public cloud This model is designed for open general public use, thus, it is only and inevitably provisioned by a third-party vendor over the internet. Hybrid cloud This model combines the advantages of at least two specific cloud computing models by bounding them as unique entities with the use of standarised technology to facilitate portability. Virtual private cloud This deployment model distinguishes itself in terms of a private constellation inside a public environment and makes other than the private cloud use of virtual private networks (VPNs). However, the advantages are the on-demand use of pooled capabilities similar to the public cloud. Source: Mell and Grance (2011), Singh et al. (2016) A review of testing cloud security 3 381 Security Today, cloud computing is omnipresent in every aspect of businesses. Thus, it is essential to be aware of security concerns in cloud surroundings as they are major challenges faced by companies (Ali et al., 2015). The security of stored data and cloud environments underlies three key IT principles: availability, integrity and confidentiality (Ardagna et al., 2014). Availability in cloud environments is primarily affected by external attacks. Vendors backup services and sustainability are most relevant (Chen and Zhao, 2012). Integrity in cloud surroundings pertains data and computing alike. Computing integrity is the accurate execution of software without any alteration of other detrimental influences. Since clients have no authority of utilised servers, vendors might deploy insecure and outdated source code (Xiao and Xiao, 2013). Zafar et al. (2017) proposed the usage of data integrity schemes to timely identify data corruption or deletion fostering undertaking actions for data recovery. On the other hand, data integrity depicts the unaltered condition and if so the detection of modifications. Data stored on cloud computing servers might be falsely administered or modified. Assailants could particularly target the loss of data control to take any advantages. Confidentiality of data must consider encryption algorithms and key strengths. Otherwise, confidentiality of clouds depicts the confidential keeping of data from both contractors as well as customers computing tasks. Though, the multi-tenancy characteristic is likely to cause breaches in privacy and data confidentiality owing to lacks of strong customer authentication operated on the same platform, using similar resources (Zissis and Lekkas, 2012). 3.1 Cloud security Generally, the adoption and use of cloud services go along with handing over the authority of data into the hands of a third party aggravated by the access of many users over the internet (Ali et al., 2015). Figure 5 shows the cloud security reference model, which clarifies the actors, operations, entities and infrastructure in cloud environments. Figure 5 Cloud security reference model (see online version for colours) Source: Fernandes et al. (2014) Concisely, CSPs offer, in their own data centres hosted, services to clients. To ensure confidentiality, the provider has to limit access to only authorised customers. Especially in public cloud environments are the two parties connected via the Internet. As so, CSPs need to guarantee the integrity by preserving information content. Conditions of provisioned services, such as the availability of data, are determined in SLAs between CSPs and customers. 382 E. Zenker and M. Shahpasand Cloud security risks and measures Table 4 Category Data security and privacy Risk Security measures Ensures availability of • Specific security measures have been taken by customers data in the cloud CSPs to prevent outages and attacks. Risk related to data security and privacy • To mitigate these risks APIs are used to implement a robust access control via encryption to protect data traffic. • Analysis of protected data during design time and runtime. • Provide effective mechanisms for key generation, storage, and destruction of data. Preventing unauthorised access to customers data in the cloud • Can be resolved by implementing identity management, authentication and authorisation techniques on both customers and providers aides. Risk related to multi-tenancy • CSP should use effective encryption methods to guarantee data isolation between clients. Risk related to data deletion • The provider should define policies to establish procedures for the destruction of persistent media before throwing it out. Lack of standardised technology in the cloud computing system • The customer should ensure if the provider uses standardised technology and it should be mentioned in its initial contract. Compatibility issues between cloud and IT systems in the customer’s organisation • The solution is to use the hybrid cloud, which is capable of handling much of these compatibility issues. Risk related to resource planning, change management. • Involves stakeholders in the cloud adoption procedures. Risk related security management • Re-evaluate existing security standards before the adoption of clouds. Physical security The physical security of cloud providers data centres composed of servers, storage and network devices. • Cloud providers must have certain policies and procedures in place to prevent physical security breaches. These include physical location security like alarms, CCTV cameras etc. Compliance Enforce regulatory obligations in a cloud environment. • The CSP must abide by all the regulations include HIPPA, FISMA. Business continuity and disaster recovery Recommends replicating data across multiple infrastructures to avoid vulnerabilities in the event of a major failure. Technology Organisational Source: Latif et al. (2014) • The CSP has to contend with the legal systems under different jurisdiction with not much of visibility as to where the data resides and how it is routed by passing through different legal jurisdictions. A review of testing cloud security 383 Recently, standards regarding CSP interoperability to foster the homogenous and consistent migration between various cloud models have been established, such as open cloud consortium (OCC), IEEE cloud computing standard study group (IEEE CCSSG) or CSA (Rong et al., 2013). Though, none of these standards are generally and widely accepted by the industry as a common approach owing to different focus areas and interests of the organisations. Moreover, those approaches lack defined guidelines for compatibility and free data movement among clouds. As a result, no inter-cloud standard has been established (Rong et al., 2013). Thus, the NIST is going to introduce a cloud computing security reference architecture (NIST Cloud Computing Security Working Group, 2013). As a consequence of missing standards, the industry applied the use of SLAs for specific and direct determinations between a provider and a customer. SLAs are widely known, accepted and used for classical outsourcing. Nevertheless, these agreements generally just specify minimum levels and until last did not consider confidentiality and integrity (Rong et al., 2013). On the downside, everything SLAs do not cover triggers disputes about consequences in case of counteractive measures and breaches. Furthermore, other than coherent standards, SLAs are pre-defined, not negotiable contracts and vary from vendor to vendor (Ali et al., 2015). The cloud security reference model also exemplifies external and internal risks. Table 4 lists some security risks and measures to consider. 3.2 Vulnerabilities Chou (2013) classified cloud vulnerabilities within three categories. According to his taxonomy, is the underlying infrastructure including virtualisation a major vulnerability. Second, comes the access of cloud services and stored data. Lastly, he depicts the medium, mostly the internet. Moreover, the author points out that vulnerabilities can occur from inside and outside of cloud environments. Khalil et al. (2014) extended the taxonomy and breaks it down more detailed within five categories: C1 Security standards are required to take precaution measures in cloud computing in order to prevent attacks. It governs the policies of cloud computing for security without compromising reliability and performance. C2 The network category refers to the medium through which the users connect to cloud infrastructure to perform the desired computations. It includes browsers, network connections and information exchange through registration. C3 The access control category covers authentication and access control. It captures issues that affect privacy of user information and data storage. C4 The data category cover data related security issues including data migration, integrity, confidentiality, and data warehousing. C5 The cloud infrastructure category includes security issues within SaaS, PaaS and IaaS and is particularly related with virtualisation environment. 384 Table 5 E. Zenker and M. Shahpasand Cloud security classifications and issues No. Category C1 Security standards Issue • Lack of security standards (I1) • Compliance risks (I2) • Lack of auditing (I3) • Lack of legal aspects (SLA) (I4) • Trust (I5) C2 Network • Proper installation of network firewalls (I6) • Network security configurations (I7) • Internet protocol vulnerabilities (I8) • Internet dependence (I9) C3 Access control • Account and service hijacking (I10) • Malicious insiders (I11) • Authentication mechanism (I12) • Privileged user access (I13) • Browser security (I14) C4 Data • Data redundancy (I15) • Data loss and leakage (I16) • Data location (I17) • Data recovery (I18) • Data privacy (I19) • Data protection (I20) • Data availability (I21) C5 Cloud infrastructure • Insecure interface of API (I22) • Quality of service (I23) • Sharing technical flaws (I24) • Reliability of suppliers (I25) • Security misconfiguration (I26) • Multi-tenancy (I27) • Server location and backup (I28) Source: Khalil et al. (2014) Furthermore, Khalil et al. (2014) mapped specific security issues to each category in Table 5. There are various standards like from organisations like the CSA and the OCC available but the industry could not agree to a common due to different interests of enterprises and institutions (Rong et al., 2013). However, in case of applying a standard, multiple security concerns are associated with compliance perils owing to deficiencies of assessments and audits of corporate standards. Besides, differing legal laws, rules and regulations concern cloud parties. In relation is trust, where cloud customers struggle with legal claims in terms of data breach or loss. Likely overlooked proper security A review of testing cloud security 385 configurations and installations of firewalls within cloud networks facilitate the access of hackers. Hackers may also identify vulnerabilities within internet protocols and use them to intrude the network. Once they managed to enter the network, the internet connection can be unavailable and the cloud service not be accessed. Another concern is the unauthorised access, which can be realised from outside or inside of an organisation. Especially, administrators have a lot of power and authority to manipulate services. Furthermore, weak authentication mechanisms in combination with privileged single access from various platforms can lead to intruding other services. Mostly, the platforms are web browsers, which contain several vulnerabilities facilitating unauthorised access. Data availability, protection, privacy, recovery, location and loss are major security issues. In case data is not appropriately controlled, protected, transmitted and encrypted, it is nothing to intercept them. In terms of the infrastructure, vulnerable APIs in the cloud portal expose an enterprise to logging capabilities, reusable tokens, content transmission and unauthorised access. Outsourced activities, such as maintaining hardware and servers, facilitate the access of untrusted suppliers. More vulnerabilities can occur in misconfigured platforms, custom codes, web servers, frameworks and the application stack (Khalil et al., 2014). The infrastructure of cloud environments is distributed around the globe. A CSP shares the infrastructural and technical resources among its clients to maximise efficiency and performance via resource pooling. This structure is vulnerable to cross-tenant attacks. In detail, the CSP uses virtual networks to dedicate the pooled resources to a particular client. Thus, these networks exacerbate security mechanisms and physical protection to detect possible threats and monitor traffic (Ali et al., 2015). Moreover, the fact that CSPs host multiple customers on the same platform (multi-tenancy) exploits vulnerabilities in terms of privacy breaches due to shortages of strong authentication (Zissis and Lekkas, 2012). 3.3 Threats Based on the identified cloud vulnerabilities, presents the author applicable attacks and sample incident scenarios. Table 6 shows a summary of known attacks, their consequences and the exploited vulnerabilities. Theft-of-service attacks use scheduler vulnerabilities of hypervisors. The assault is conducted during a scheduling mechanism of the hypervisor that fails to detect the central processing unit (CPU) use of poorly deployed VMs. This can lead to unauthorised access of clients. However, it is mostly relevant to public cloud environments as clients are charged by the runtime of their VMs rather than by the runtime of the CPU. One of the biggest threats are DoS attacks because they are considerably easy to implement and difficult to countermeasure by security experts. Especially, the use of XML and HTML is critically vulnerable. A cloud customer initiates a XML request and sends following the request via the HTML protocol trough the system interface. Exactly this system interface is the point of failure due to undetected vulnerabilities. A malware injection assault denotes altered copies of service instances, that are uploaded to the cloud. As a result, victim’s service requests are handled within the malicious instance leading to access to personal user data. A cross VM side channel attack uses the circumstance that VMs use the same physical hypervisor platform and thus, share hardware resources. As so, a malicious VM can access cache locations of other virtual instances to infer the victims behaviour. One step further go targeted shared memory attacks. They utlise shared 386 E. Zenker and M. Shahpasand memory advantages, not only of the VMs but also of physical. This is one way of how malware injection attacks can be conducted. Phishing attacks attempt to gain personal information via compromised websites, emails, etc. As so, attackers can acquire login data and user credentials which can be done in two ways. First, the attackers emulate the cloud service website and second, hijacking account data in traditional scams. Botnet or stepping-stone attacks aim to disguise the identities and locations of the attackers to exacerbate the backtrace. This is done by an indirect attack through a sequence of other hosts. Most of the time, the host do not know that they are a part of a botnet through unnoticed infiltration of malware. When conducting audio steganography attacks, attackers hide their private data within media files. They appear to be usual, unsuspicious files by deceiving security mechanims and countermeasures. Lastly, in VM rollback attacks are snapshots of current memory and disk usage as well as CPU states used. In this scenario, the attacker uses previous snapshots without user notice to clear the history. As so, the suspicious activities cannot be caught. An example is brute force attacks to guess the login credentials. In case the guest operating system (OS) is restricted to specific number of attempts, the attacker can rollback the VM and start over again (Khalil et al., 2014). Table 6 No. 1 Cloud attacks Attack Incidents Theft-of-service Consequences Category Caused by • Cloud services usage without billing C5 I1, I3, I6, I8, I11, I14, I26 C2, C5 I1, I3, I10, I14, I26 C5 I7, I11, I13, I22 C5 I22, I26 • Cloud resource stealing with less/no cost 2 Denial of service DDoS, • Service hardware HTTP-based DDos, unavailability XML-based DDos, REST-based DDoS, • Wrapping a malicious code in XML Shrew attack (light signature to gain traffic) DDoS unauthorised access to information • Accessing a browser history or any other private information through unsecure HTTP browsing 3 • Credential information leakage Cloud malware injection • User data leakage • Cloud machine abnormal behaviour 4 Cross VM side channels Timing side channels, energy consumption side channels Source: Khalil et al. (2014) • User data/information leakage • Cloud resources/infrastructur e information leakage 387 A review of testing cloud security Table 6 No. 5 Cloud attacks (continued) Attack Incidents Targeted shared memory Consequences Category Caused by • Cloud resource’s information leakage C5 I1, I3, I10, I22, I26 C2, C3, C5 I1, I6, I8, I10, I12, I14 C2, C3, C5 I1, I6, I10, I12, I14 C3, C5 I1, I3, I6, I10, I14, I26 C3, C5 I1, I3, I6, I10, I14, I26 • User information/data leakage • Provides open window for other attacks such as side channels and cloud malware injection 6 • Unauthorised access to personal information Phising • Installing a malicious code into user computer • Force cloud computing structure to behave abnormally • Make server unavailable for end-user. 7 Bonets Stepping stone attack • Unauthorised access to cloud resources • Make cloud system work abnormally • Stealing sensitive information • Stealing user data 8 Audio stegonography • Unavailability of cloud storage system • Accessing user data • User data deletion 9 VM rollback attack • Launch brute force attack • Damage cloud infrastructure • Leakage of sensitive information Source: Khalil et al. (2014) According to Modi et al. (2013), major CSPs have adopted several security measures. For example, Amazon is using SSL encryption for its simple storage service. In addition, 388 E. Zenker and M. Shahpasand Google and Microsoft use sandbox environments to isolate specific, e.g., Java, applications from others. Microsoft also deploys firewalls, router filters and security patches. Moreover, Salesforce.com applied the security assertion markup language (SAML) as a user authentication mechanism. 4 Testing Felderer et al. (2016) stated that testing is the evaluation of systems via the observation of its execution. This system is titled system under test (SUT). According to the International Software Testing Qualifications Board (ISTQB, 2011a), is software testing an execution process to validate and verify programs and applications. Moreover, ISTQB defined a general guideline, including seven principles of software testing as shown in Table 7. Table 7 No. Software testing principles Principle Description 1 Testing shows presence of defects Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness. 2 Exhaustive testing is impossible Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive texting, risk analysis and priorities should be used to focus texting efforts. 3 Early testing To find defects early, testing activities shall be started early as possible in the software or system development life cycle, and shall be focused on defined objectives. 4 Defect clustering Testing effort shall be focused proportionally of the expected and later observed defect density of modules. A small number of modules usually contains most of the defects discovered during prerelease testing, or is responsible for most of the operational failures. 5 Pesticide paradox If the same tests are repeated over and over again, eventually the same set of test cause will no longer find any new defects. To overcome this ‘pesticide paradox’, test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to find potentially more defects. 6 Testing is context dependent Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site. 7 Absence-of-errors fallacy Finding and fixing defects does not help if the system built is unusable and does not fulfil the users’ needs and expectations. Source: ISTQB (2011a) The International Organization for Standardization (ISO, 2014) states in the 25,000 standard six quality criteria for the evaluation of software: • maintainability • usability A review of testing cloud security • efficiency • portability • reliability • functionality. Figure 6 389 Black-box test design Furthermore, test types are divided into static and dynamic tests. Static tests comprise reviews and analysis of source code. On the contrary, dynamic tests are based on the execution, observation and evaluation of the software component. Based on the above introduced criteria, derived several dynamic test design techniques (ISTQB, 2011a). Functional tests are based on functions, characteristics and their interoperability to other software components and systems. This scenario considers the observable performance without any knowledge of the internal structure. This is also known as black-box testing. Figure 6 illustrates the design of a black-box test. Contrarily, a structure-based or white-box test is based on the source code and interfaces as visualised in Figure 7. Test cases are designed with the knowledge of programme sequences. Non-functional tests focus on the overall performance, execution time, load, etc. Regression tests repeat trial executions of already tested parts after modifications (ISTQB, 2011b). Figure 7 White-box test design 4.1 Cloud testing Cloud testing is the intersection of computer programs as well as cloud testing (Kumar and Singh, 2014). More in detail, it is an evaluation form where tested applications use 390 E. Zenker and M. Shahpasand cloud capabilities to simulate realistic use cases. Moreover, it aligns SaaS and cloud concepts. The goal is to assure the delivery of high-quality services (Nachiyappan and Justus, 2015). Besides, cloud testing is classified in two major categories i.e., TaaS and testing the cloud. TaaS or cloud-based tests provide on-demand cloud capabilities such as tools and computing power to execute functional as well as non-functional tests. On the other hand, testing the cloud validates and verifies the on-demand offered services to clients (Asif et al., 2015). Figure 8 depicts a comprehensive overview of considerable cloud dimensions to test. 4.2 SaaS testing Nowadays, clients and providers of SaaS have a very strong commitment to the quality of service (QoS). According to Gao et al. (2013b), stated Salesforce.com import quality measures as trusted security, massive scalability, high on-demand availability, maximum performance and uptime as well as reliability. Most of those requirements are defined in SLAs between both parties. SaaS testing comprises validation activities of applications in a test procedure to ensure QoS. This includes underlying infrastructures and networks as well (Gao et al., 2013; Prakash et al., 2012). Table 8 shows the main SaaS testing tasks and objectives. There is an intersection to classic software testing, but unlike conventional approaches, SaaS testing needs to cover scalability and elasticity as well as load generation and large-scale test cases. Table 8 SaaS testing tasks and objectives SaaS testing task Objectives and focuses Component testing Perform black-box and white-box testing for components Function testing Test tenant-based service functions, behaviours, workflows and transactions Integration testing Perform integration between SaaS systems and others. Deployment and recovery Test SaaS deployment and it fault-recovery Check multi-tenant based service integration Multi-tenancy testing Test multi-tenant-based functions and services Quality-of-service (QoS) Assure the given QoS requirements in SLA agreements, including scalability, reliability, availability, performance and system throughput On-demand testing and simulation On-demand large-scale test generation and simulation Security testing Assure single/multiple tenant-based SaaS security in databases, workflows, transactions and functions. Assure user privacy and system security of SaaS. Customisation and configuration testing Assure the quality of tenant-based customisations and configurations in SaaS databases, workflows, user interfaces and functional services. Connectivity testing Assure the quality of SaaS connectivity APIs. User interface portability, and compatibility Test user interfaces in usability, portability and compatibility. Continuous upgrade testing Validate continuous upgrades of SaaS whenever new tenants are added, and/or existing software is changed. Source: Gao et al. (2013) A review of testing cloud security 391 4.3 Cloud security testing Figure 8 points security testing in cloud surroundings as one essential cloud test dimension. Security testing validates application requirements regarding security properties. It identifies if the specified security features are correctly implemented. Security testing is distinguished in functional testing and vulnerability exposure. The former validates the correct implementation of specific security requirements regarding mechanisms and properties. The latter tries to identify yet unrevealed application vulnerabilities (Felderer et al., 2016). Among the main challenges in cloud security testing is the characteristic of multi-tenancy. This is causing a security concern called traversal vulnerability. One tenant could traverse from a virtual machine (VM) to another one on the same hypervisor. As so, a tenant is able to access a virtual instance of other clients. Moreover, multi-tenancy requires penetration testing, checking against structured query language (SQL) injections, cross-site scripting (XSS) and uniform resource locator (URL) manipulation. The test needs to be executed by a malicious user and valid credentials to the SUT and the underlying database (Nachiyappan and Justus, 2015). Some attacking scenarios are DoS, IP spoofing, and man in the middle. In this case, networks or servers are brought down by huge traffic. This traffic can be generated via hacking tools, delays of network packets or congestion. Another aspect to be tested is the identity federation management. Moreover, security testing should ensure that cookie values are encrypted and the application contains no hidden form fields (Vemulapati et al., 2011). A single sign-on (SSO) mechanism is used for users to only log in once and access multiple system components without being prompted for his credentials again. Other security domains to be checked include the application development user interface, data management and role-based access control (Iyer, 2016). Figure 8 Cloud test dimensions (see online version for colours) Source: Iyer (2016) 392 5 E. Zenker and M. Shahpasand Related work Halabi and Bellaiche (2017) proposed a method for evaluation and performance quantification of cloud security services. By the use of the goal-question-metric paradigm, the researchers developed quantitative evaluation metrics applying them to a case study to demonstrate the practicability and efficiency. This method is designed and proposed for CSPs whereas the paper points out that it can be easily automated on cloud customer sides. Albonico et al. (2016) focused on elastic testing of cloud applications during various elasticity states. The authors proposed a procedure for test executions based on monitoring. This approach comprises status monitoring of resources to recognise occurrences at real-time and different elasticity states. By the use of experimental test cases, the researchers identified non-functional errors and validated their procedure. However, the proposed procedure is of a simple nature and does not consider securityrelevant factors. Riungu-Kalliosaari et al. (2016) conducted a qualitative study on the adoption cloud computing testing in an organisational context. The authors applied the grounded theory on, via interviews collected, data. The researchers concluded that cloud-based capabilities could facilitate organisations testing needs. Moreover, cloud-based testing will improve the final products. However, this survey focuses only on the client side of cloud services. Chang et al. (2016) presented a three-layered security framework for business clouds based on encryption, identity management and firewalls. The authors evaluated their approach through penetration tests where 99.95% of Trojans and viruses were blocked and detected. In addition, the adoption of this framework can block all SQL injections. Felderer et al. (2016) provided a taxonomy for model-based security tests through a thorough literature review. This approach is based on classification schemes for security and model-based testing. Furthermore, it consists of security-specific filter and maturity evidence criteria. This taxonomy is not specifically proposed for cloud solutions. Kiran et al. (2014) proposed a similar framework suitable for cloud environments. Anisetti et al. (2015) presented a testbed assurance scheme to incrementally certify cloud security. Hence, the scholars defined assurance techniques for increasing cloud transparency. The suggested scheme lowers costs via reusing existing certificates. Fernandez et al. (2015) proposed a methodology to build a security reference architecture for clouds with the use of unified modelling language (UML) models. For this purpose, the authors presented misuse and security patterns as well as a metamodel. First, the scholars identified cloud security threats and described likely attacking scenarios. Subsequently, the evaluation was done by building a catalogue of newly developed cloud misuse patterns and comparing the architecture with the NIST approach. The findings showed that their model represented all security features and thus, was more precise. Nevertheless, this approach is more of a general manner and does not describe implementations ways. Asif et al. (2015) presented a framework for performance testing for small as well as large SaaS cloud applications on the basis of the performance testing lifecycle. This general approach can be used for any SaaS application. Moreover, performance testing comprised scalability, stress and load testing in combination with the performance metrics throughput, availability and response time. The researchers evaluated their proposal via questionnaires answered by experts. This study lacks evaluation results whereby the proposed framework cannot be entirely approved. A review of testing cloud security 393 Hosseini et al. (2015) proposed a cloud testing framework on the basis of ISTQB standards. This approach comprised test scenario development, design of test cases, cloud provider selection, infrastructure setup, cloud server leverage, test start, test progress monitoring, test report and closure. The authors evaluated their framework against the C-Meter framework resulting in many advantages. Nevertheless, it is an abstract model, which needs to be more drilled-down. Oliveira et al. (2015) introduced an approach for test case creation for SUTs in cloud environments based on a model transformation of model-driven engineering (MDE). Model-driven testing supported the test of generated source code from model transformation. Moreover, particular testing criteria were introduced within testing metamodels. Patel and Shah (2015) proposed a combined approach of the benefits of variability modelling and MDE for automated SaaS testing. The test cases were modelled via the use of the enterprise software test modelling language. Moreover, the common variability language was used to model variability. From those were automated test scripts generated. These test cases resulted from a model transformation. The experimental evaluation revealed that the modelling of certain variations takes considerably longer. However, this approach is more sophisticated than the proposed one of Oliveira et al. (2015). Whaiduzzaman and Gani (2014) proposed an automated software scripting model by penetration testing on the cloud vendor side. Furthermore, the researchers identified CSP vulnerabilities and checked the strength of security as well as the fault tolerance. Employing their findings, the scholars defined metrics to rank the trustworthy of CSPs. However, the defined metrics are non-measurable like location, customer satisfaction etc. Kiran et al. (2014) presented a certification strategy of SaaS solutions to automate tests. The basis of this approach is the adoption of functional and state-based specifications. This standard method supports the certification of the service life cycle process. The extensible markup language (XML) specification language was used to design practical test cases. The evaluation revealed the exposure of non-obvious flaws in a SAP HANA case study. Other than Felderer et al. (2016) is this model-based framework suitable for cloud environments. Narula and Sharma (2014) introduced a framework for analysing and testing cloud services. This framework consists of a series of planned tasks and is more a cloud testing lifecycle. The authors state that the adoption of this framework reduces the test execution time of large test cases and leads to enhanced cost efficiency without any evaluation of its performance nor a proof or justification for this statement. The proposed steps are similar to those from Hosseini et al. (2015). Zhou et al. (2014) introduced a template-based approach to generate test cases and scripts automatically for service performance measures in private PaaS enterprise clouds. Additional to this PaaS approach, Asif et al. (2015) developed a similar one for SaaS applications. This empirical research revealed potential performance issues and significant cost savings for performance testing. However, this approach is not applicable for common web services. Neto and Garcia (2013) developed an integrated cloud-testing framework, including tools, techniques, roles and activities by using a mapping approach. This model standardises testing in cloud environments to ensure the quality of cloud services. 394 E. Zenker and M. Shahpasand Riungu-Kalliosaari et al. (2012) proposed a practical roadmap for the adoption of cloud-based testing. The scholars approached their study by conducting interviews in software organisations. The roadmap comprised steps to identify specific benefits for the organisation and then firstly conduct pilot projects to further elaborate strategies. This study is limited by the focus on the client side of cloud services. Zech et al. (2012) suggested a change-driven and model-based security testing approach among all layers in a cloud environment. The risk-driven changes were generic and expandable in terms of vulnerabilities. The scholars experimentally evaluated their approach. However, this approach did not assure a system’s validity, but it could show its deficiencies. Jenkins et al. (2011) suggested an approach for testing cloud infrastructures and platforms. This intelligent framework accelerated testing and was capable of developing test cases simultaneously. The evaluation was done using case studies. Nevertheless, this approach is a cloud program that makes use of plugins to test APIs. Vemulapati et al. (2011) described how software testing principles can be applied to SaaS applications within a framework. The proposed SaaS security model evaluates, in this paper, identified security standards for any SaaS solution. However, this approach is of a very general manner. 6 Conclusions and future work The cloud computing paradigm is a nascent technology with many benefits for organisations. On the other side, security of clouds is still one of the major concerns of clients to adopt and use the new computing paradigm. The review of the recent academic literature revealed that cloud security in general is still a major concern in the industry and academics alike. To make it clearer, a data breach revealed vulnerabilities at Yahoo! Inc., whereby 32 million user accounts were accessed by forged cookies to log in without a password (Yahoo! Inc., 2017). Another example is the distributed denial-of-service (DDoS) attack against Dyn, which was just recently acquired by Oracle, causing a major breakdown of its domain name system (DNS) servers also affecting enterprises relying on SaaS (York, 2016). Besides, the survey unfolded that security and related testing activities are current research fields with a lot of open queries. Thus, many academic papers have been published to identify and address challenges in cloud security, vulnerabilities and threats. However, most of the researchers focused on TaaS rather than on testing the cloud. Hence, this survey reveals a current gap in academic research in terms of testing the cloud security using an appropriate approach for SaaS applications. The authors imply to conduct further research on cloud security testing approaches, especially in SaaS and public environments, whereby internal and external factors need to be differentially considered. A review of testing cloud security 395 References Akhgar, B. (2016) ‘Our combined vision for the future of cybercrime research’, in Current and Emerging Challenges in Cybercrime and Cyberterrorism, p.36, Centric, Den Haag, NL. Alam, M.I., Pandey, M. and Rautaray, S.S. (2015) ‘A comprehensive survey on cloud computing’, International Journal of Information Technology and Computer Science (IJITCS), Vol. 7, No. 2, pp.68–79. Albonico, M., Mottu, J-M. and Sunyé, G. (2016) ‘Monitoring-based testing of elastic cloud computing applications’, in ICPE Companion (LT Workshop), ACM, Delft. Ali, M., Khan, S.U. and Vasilakos, A.V (2015) ‘Security in cloud computing: opportunities and challenges’, Information Sciences, Vol. 305, pp.357–383. Anisetti, M., Ardagna, C.A. and Damiani, E. (2015) ‘A test-based incremental security certification scheme for cloud-based systems’, in 2015 IEEE International Conference on Services Computing (SCC), IEEE, pp.736–741. Ardagna, C.A., Asal, R., Damiani, E. and Vu, Q.H. (2014) ‘From security to assurance in the cloud: a survey’, ACM Computing Surveys (CSUR), Vol. 48, No. 1, p.2. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I. and Zaharia, M. (2010) ‘A view of cloud computing’, Communications of the ACM, Vol. 53, No. 4, pp.50–58. Asif, S., Rehman, M., Anjum, M. and Saleemi, F. (2015) ‘Framework for testing cloud base applications’, Bahria University Journal of Information & Communication Technology, Vol. 8, No. 2, pp.75–83. Bos, H., Etalle, S. and Poll, E. (2014) National Cyber Security Research Agenda – Trust and Security for our Digital Life. Buyya, R., Yeo, C.S. and Venugopal, S. (2008) ‘Market-oriented cloud computing: vision, hype, and reality for delivering IT services as computing utilities’, in 10th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGrid 2010), IEEE, Melbourne, p.9. Chang, V., Kuo, Y-H. and Ramachandran, M. (2016) ‘Cloud computing adoption framework: a security framework for business clouds’, Future Generation Computer Systems, Vol. 57, pp.24–41. Chen, D. and Zhao, H. (2012) ‘Data security and privacy protection issues in cloud computing’, Computer Science and Electronics Engineering (ICCSEE), 2012 International Conference, Vol. 1, pp.647–651. Chou, T-S. (2013) ‘Security threats on cloud computing vulnerabilities’, International Journal of Computer Science & Information Technology (IJCSIT), Vol. 5, No. 3, pp.79–88. Christophe, C., Coti, C., Delort, P., Diaz, F., Gagnaire, M., Mijic, M., Gaumer, Q., Guillaume, N., Lous, J. Le, Lubiarz, S., Raffaelli, J-L., Shiozaki, K., Schauer, H., Smets, J-P., Laurent, S. and Ville, A. (2014) Downtime Statistics of Current Cloud Solutions. Felderer, M., Zech, P., Breu, R., Büchler, M. and Pretschner, A. (2016) ‘Model-based security testing: a taxonomy and systematic classification’, Software Testing, Verification and Reliability, Vol. 26, No. 2, pp.119–148. Fernandes, D.A.B., Liliana, S.F.B., Gomes, J.V, Freire, M.M. and Inácio, P.R.M. (2014) ‘Security issues in cloud environments: a survey’, International Journal of Information Security, Vol. 13, No. 2, pp.113–170. Fernandez, E.B., Monge, R. and Hashizume, K. (2015) ‘Building a security reference architecture for cloud systems’, Requirements Engineering, Vol. 21, No. 2, pp.225–249. 396 E. Zenker and M. Shahpasand Gao, J., Xiaoying, B., Tsai, W.T. and Uehara, T. (2013) ‘SaaS testing on clouds – issues, challenges, and needs’, in 2013 IEEE Seventh International Symposium on Service-Oriented System Engineering, IEEE, pp. 409–415. Halabi, T. and Bellaiche, M. (2017) ‘Towards quantification and evaluation of security of cloud service providers’, Journal of Information Security and Applications, pp.1–11. Hogan, M., Liu, F., Sokol, A. and Tong, J. (2013) NIST Cloud Computing Standards Roadmap. Hosseini, S., Nasiri, R. and Shabgahi, G. (2015) ‘A new framework for cloud based application testing’, International Journal of Scientific Engineering and Applied Science (IJSEAS), Vol. 1, No. 3, pp.112–118. International Organization for Standardization (ISO) (2014) Systems and Software Engineering – Systems and Software Quality Requirements and Evaluation (SQuaRE) – Guide to SQuaRE, ISO/IEC 25000:2014. International Software Testing Qualifications Board (ISTQB) (2011a) Certified Tester Foundation Level Syllabus. International Software Testing Qualifications Board (ISTQB) (2011b) Standard Glossary of Terms Used in Software Testing. Iyer, G.N. (2016) ‘Cloud testing: an overview’, in Murugesan, S. and Bojanova, I. (Eds.): Encyclopedia of Cloud Computing, pp.327–337, Wiley, Chichester. Jenkins, W., Vilkomir, S., Sharma, P. and Pirocanac, G. (2011) ‘Framework for testing cloud platforms and infrastructures’, in 2011 International Conference on Cloud and Service Computing, IEEE, pp.134–140. Jula, A., Sundararajan, E. and Othman, Z. (2014) ‘Cloud computing service composition: a systematic literature review’, Expert Systems with Applications, Vol. 41, No. 8, pp.3809–3824. Khalil, I.M., Khreishah, A. and Azeem, M. (2014) ‘Cloud computing security: a survey’, Computers, Vol. 3, No. 1, pp. 1–35. Kiran, M., Friesen, A. and Simons, A.J.H. (2014) ‘Model-based testing in cloud brokerage scenarios’, in International Conference on Service-Oriented Computing, pp.192–208, Springer International Publishing, Berlin. Ko, R. and Lee, S.S.G. (2013) Cloud Computing Vulnerability Incidents: A Statistical Overview. Kumar, R. and Singh, S. (2014) ‘Cloud testing: perspective and challenges’, International Journal of Computer Applications, Vol. 106, No. 17. Latif, R., Abbas, H., Assar, S. and Ali, Q. (2014) ‘Cloud computing risk assessment: a systematic literature review’, Future Information Technology, pp.285–295. Madan, M., Dave, M. and Tandon, A. (2016) ‘Challenges in testing of cloud based application’, International Journal of Advanced Research in Computer Science and Electronics Engineering (IJARCSEE), Vol. 5, No. 1, pp.28–31. Majendran, S. (2013) Organizational Challenges in Cloud Adoption and Enablers of Cloud Transition Program, Cambridge, MA. Mell, P. and Grance, T. (2011) The NIST Definition of Cloud Computing: Recommendations of the National Institute of Standards and Technology, Gaithersburg, MD. Modi, C., Patel, D., Borisaniya, B., Patel, A. and Rajarajan, M. (2013) ‘A survey on security issues and solutions at different layers of cloud computing’, The Journal of Supercomputing, Vol. 63, No. 2, pp.561–592. Nachiyappan, S. and Justus, S. (2015) ‘Cloud testing tools and its challenges: a comparative study’, Procedia Computer Science, Vol. 50, pp.482–489. Narula, E.T. and Sharma, E.G. (2014) ‘Framework for analyzing and testing cloud based applications’, International Journal of Advanced Research in Computer Science and Software Engineering, Vol. 4, No. 6, pp.592–596. Neto, C.R.L. and Garcia, V.C. (2013) ‘Cloud testing framework’, in Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, ACM, pp.252–255. A review of testing cloud security 397 NIST Cloud Computing Security Working Group (2013) NIST Cloud Computing Security Reference Architecture. Oliveira, J., Lopes, D., Abdelouahab, Z., Claro, D. and Hammoudi, S. (2015) ‘Model driven testing for cloud computing’, Innovations and Advances in Computing, Informatics, Systems Sciences, Networking and Engineering, pp.297–304. Patel, S. and Shah, V. (2015) ‘Automated testing of software-as-a-service configurations using a variability language’, in Proceedings of the 19th International Conference on Software Product Line, ACM, pp.253–262. Prakash, V., Ramadoss, R. and Gopalakrishnan, S. (2012) ‘Software as a service (SaaS) testing challenges – an in-depth analysis’, IJCSI International Journal of Computer Science Issues, Vol. 9, No. 3, pp.506–510. RightScale Inc. (2016) State of the art Cloud Report. Riungu-Kalliosaari, L., Taipale, O. and Smolander, K. (2012) ‘Testing in the cloud: exploring the practice’, IEEE Software, pp.46–51. Riungu-Kalliosaari, L., Taipale, O., Smolander, K. and Richardson, I. (2016) ‘Adoption and use of cloud-based testing in practice’, Software Quality Journal, Vol. 24, No. 2, pp.337–364. Rong, C., Nguyen, S.T. and Jaatun, M.G. (2013) ‘Beyond lightning: a survey on security challenges in cloud computing’, Computers and Electrical Engineering, Vol. 39, No. 1, pp.47–54. Ruan, K., Carthy, J., Kechadi, T. and Baggili, I. (2013) ‘Cloud forensics definitions and critical criteria for cloud forensic capability: an overview of survey results’, Digital Investigation, Vol. 10, No. 1, pp.34–43. Shrivastva, A., Shubham, G. and Rinki, T. (2014) ‘Cloud based testing techniques (CTT)’, International Journal of Computer Applications, Vol. 104, No. 5, pp.24–29. Singh, S., Jeong, Y. and Park, J.H. (2016) ‘A survey on cloud computing security: issues, threats, and solutions’, Journal of Network and Computer Applications, Vol. 75, pp.200–222. Statista Inc. (2016) Worldwide Software as a Service (SaaS) Revenue from 2010 to 2016 (in Billion U.S. Dollars) [online] https://www.statista.com/statistics/273642/worldwide-software-as-aservice-revenue-forecast/ (accessed 19 January 2017). Tao, L., Xiaojun, Y. and Jianmin, W. (2012) ‘Protecting data confidentiality in cloud systems’, in Proceedings of the Fourth Asia-Pacific Symposium on Internetware, ACM, p.18. Vemulapati, J., Mehrotra, N. and Dangwal, N. (2011) ‘SaaS security testing: guidelines and evaluation framework’, in 11th Annual International Software Testing Conference 2011. Whaiduzzaman, M. and Gani, A. (2014) ‘Measuring security for cloud service provider: a third party approach’, in 2013 International Conference on Electrical Information and Communication Technology (EICT), IEEE, pp.1–6. Xiao, Z. and Xiao, Y. (2013) ‘Security and privacy in cloud computing’, IEEE Communications Surveys & Tutorials, Vol. 15, No. 2, pp.843–859. Yahoo! Inc. (2017) Annual Report [online] https://investor.yahoo.net/secfiling.cfm?filingID= 1193125-17-65791&CIK=1011006&soc_src=mail&soc_trk=ma (accessed 27 March 2017). York, K. (2016) Dyn Statement on 10/21/2016 DDoS Attack [online] http://dyn.com/blog/dynstatement-on-10212016-ddos-attack/ (accessed 27 March 2017). Zafar, F., Khan, A., Ur, S., Malik, R., Ahmed, M., Anjum, A., Khan, M.I., Javed, N., Alam, M. and Jamil, F. (2017) ‘A survey of cloud computing data integrity schemes: design challenges, taxonomy and future trends’, Computers & Security, Vol. 65, pp.29–49. Zech, P., Felderer, M. and Breu, R. (2012) ‘Towards a model-based security testing approach of cloud computing environments’, in 2012 IEEE Sixth International Conference on Software Security and Reliability Companion, IEEE, pp.47–56. Zhou, J., Zhou, B. and Li, S. (2014) ‘Automated model-based performance testing for PaaS cloud services’, in 2014 IEEE 38th International, Computer Software and Applications Conference Workshops (COMPSACW), IEEE, pp.644–649. Zissis, D. and Lekkas, D. (2012) ‘Addressing cloud computing security issues’, Future Generation Computer Systems, Vol. 28, No. 3, pp.583–592.