research-article

Open access

A Survey of Practical Formal Methods for Security

Authors:

Peter W. V. Tran-Jørgensen,

James WoodcockAuthors Info & Claims

Formal Aspects of Computing, Volume 34, Issue 1

Article No.: 5, Pages 1 - 39

https://doi.org/10.1145/3522582

Published: 05 July 2022 Publication History

PDF eReader

Abstract

In today’s world, critical infrastructure is often controlled by computing systems. This introduces new risks for cyber attacks, which can compromise the security and disrupt the functionality of these systems. It is therefore necessary to build such systems with strong guarantees of resiliency against cyber attacks. One way to achieve this level of assurance is using formal verification, which provides proofs of system compliance with desired cyber security properties. The use of Formal Methods (FM) in aspects of cyber security and safety-critical systems are reviewed in this article. We split FM into the three main classes: theorem proving, model checking, and lightweight FM. To allow the different uses of FM to be compared, we define a common set of terms. We further develop categories based on the type of computing system FM are applied in. Solutions in each class and category are presented, discussed, compared, and summarised. We describe historical highlights and developments and present a state-of-the-art review in the area of FM in cyber security. This review is presented from the point of view of FM practitioners and researchers, commenting on the trends in each of the classes and categories. This is achieved by considering all types of FM, several types of security and safety-critical systems, and by structuring the taxonomy accordingly. The article hence provides a comprehensive overview of FM and techniques available to system designers of security-critical systems, simplifying the process of choosing the right tool for the task. The article concludes by summarising the discussion of the review, focusing on best practices, challenges, general future trends, and directions of research within this field.

1 Introduction

Digital services are currently spreading to all aspects of society [207]. This in turn causes dependence of society on the cyber infrastructure needed to support these services. The heavy reliance on cyber infrastructure poses new challenges in the form of cyber attacks and potentially cyber terrorism [142], with threat actors encompassing the full range from interpersonal offenders, cyber criminals, and “hacktivists” through to well-resourced state actors [208]. Disturbances in financial, industrial, or day-to-day consumer services could lead to significant financial and societal costs. As digitisation spreads further, the potential attack surfaces only grow larger, increasing the challenge of protecting digital services [250, 260]. As systems grow larger and more complex, significant resources have to be spent to secure these system against known cyber attacks. Often the protection mechanisms are incorporated to close vulnerabilities uncovered after a successful cyber attack, and hence are of a reactive nature. This approach relegates cyber security from a primary challenge to be solved within the system to an afterthought [236].

Due to the wide spectrum of cyber attacks, it is difficult to directly quantify their impact on society [101], however, very often they involve significant financial costs as well as potential disruptions in quality of life. One example is a potential cyber attack against electricity infrastructure, including electricity marketplace, which could lead to destruction of generators and disclosure of confidential data [191]. Another example is attacks against manufacturing facilities causing delays or decrease in quality of production [47, 204]. These examples demonstrate that cyber threats should be considered as significant as physical threats against societal infrastructure.

The earlier the potential cyber security threats are discovered within new systems, the cheaper the mitigation for these threats will be [257]. Formal Methods (FM) provide an opportunity for discovery and mitigation of cyber threats at all stages of the lifecycle of a system. Using FM brings mathematical rigour to the field of cyber security assurance. This is possible, since FM are techniques that use model-based approaches, where the models are rigorously specified [261] that allow for development of precise statements about what systems under investigation should do without putting constraints on how to do it [264]. These models represent the software, hardware, or a combination of the two for the system in question. The primary benefit of using FM stems from the mathematical proof of the internal consistency of the system design [115]. This proof provides strong assurances, since it considers the entire system behaviour, and once proven true it remains true, whereas in traditional testing it is only possible to cover specific scenarios. FM can be seen as a tool well suited for providing assurances of cyber security for digital society [262]. Beyond the assurance of behavioural correctness of a system, the adoption of a fully fledged formal approach is known to reduce the number of implementation errors, which are the building blocks of exploits.

It is important to note that there exists a variety of FM. The main categories we consider are:

•

Theorem Proving, analysing a formal description for important properties based on computer-based proofs.

•

Model Checking, checking whether a finite-state model of a system meets a given specification in an exhaustive manner.

•

Lightweight FM, using formal techniques to analyse a system either statically or dynamically (this concept was coined in Reference [132], but we have extracted the model checking from their characterisation into a category of its own).

For a detailed coverage of the technical foundations of formal approaches to security, we refer the reader to Reference [32]. That chapter focuses on the technical foundations of the area, whereas our survey concentrates on specific application areas.

In all cases, the methods are applied to determine if a system behaves in a correct way, and many approaches have received significant tool support for automation of the verification and validation process [10]. In this survey, we consider all of the approaches and their application to specific areas of digital society. We further consider FM as applied to the specific level of abstraction of system behaviour ranging from the application level to the hardware level. By considering the state-of-the-art research in formal verification across these dimensions, we provide a non-exhaustive overview of application of FM in specific disciplines. The aim of this survey is to allow practitioners to identify a proven method applicable to a system in their domain, hence hopefully increasing the adoption of FM in the field of cyber security. We believe that similar surveys in different areas of use of FM could be a catalyst to increased adoption, as it has been determined that education and experience as well as finding positive examples of use of FM could lead to professionals adopting FM towards their area of expertise [104].

1.1 Methodology

The amount of research publications within the area of applying FM towards cyber security challenges is significant. Therefore, several constraints have been placed on the choice of research publications to be considered within this survey. The first important constraint is the recency of the research reported, considering the landscape of the past decade, limiting the publication date to be no earlier than 2012. Furthermore, all of the research work needs to be published in scientific venues such as journals, conferences, or workshops. The next constraint is focus on computer-based tool supported FM, i.e., only FM with tools that can provide computer-based analysis and often guide users on performing this analysis are considered. This consideration is to focus more on the FM that could be potentially applied outside of academia, bringing the benefits of the formal security analysis to industry. This goes hand-in-hand with our focus on the applied FM, searching for research publications, where a tool-supported FM is utilised to deal with a concrete cyber security problem. Hence, this survey does not focus on theoretical advances of FM in security or proposed processes that briefly mention use of FM, such as theoretical approaches to model checking algorithms, specification of hyperproperties, and similar. Furthermore, our survey does not cover the approach to security commonly referred to as provable security. This refers to a mathematical approach to analysing the security of cryptographic mechanisms or systems. The approach considers the system in the context of an attacker model and expresses the security requirements within that model as a limitation on what the attacker should be able to achieve. A proof consists of establishing that the attacker would need to break a known hard problem (such as the Quadratic Residuosity Problem [105]) to break the security of the system. Thus, the security of the system is reduced to the difficulty of the underpinning hard problem. This approach is typically used within the field of cryptography rather than secure systems, and so falls outside the scope of our survey. We point the reader to Reference [28] providing the report within the area of FM in cryptography. Finally, we constrain our search to research that considers aspects of security explicitly, and not as a by-product of safety or correctness. The search for the research publications was carried out as a cross database search using Google Scholar, while focusing on research papers, excluding research abstracts or extended abstracts.

As this survey provides readers with a quick overview of the research conducted, we have further decided to categorise different research publications by the industry (domain) on which they focus as well as the level of abstraction on which FM is utilised. In this way, researchers and also potentially industrial users can quickly find the area of their interest within this survey. Furthermore, we classify the research based on the cyber security problem classification as elicited from the discovered research papers and inspired by existing literature [208].

1.2 History

This section presents a history of impactful research works within FM in security over the past 40 years. We choose four case studies where formal methods have been applied to secure systems:

(1)

The Needham-Schroeder Public-Key Protocol. Lowe used a refinement model checker to find a triangular attack on the protocol. This was a new attack on a protocol that had previously been proven correct by Burrows et al. [52].

(2)

The Mondex smartcard. This was the first commercial product to be certified to ITSEC Level E6. There was considerable discussion at the time as to whether this was even possible.

(3)

The Tokeneer ID Station. There were similar questions about the feasibility of using FM to achieve the level of rigour required by the higher assurance levels of the Common Criteria. Tokeneer settled this matter.

(4)

The seL4 Microkernel. This system has the reputation of being the world’s most assured microkernel. Significantly, it demonstrates that security and the use of formal methods do not lead to poor performance.

The Needham-Schroeder Public-key Protocol. The Needham-Schroeder Public-Key Protocol is a transport-level protocol for communication between network devices [190], providing mutual authentication between two parties in a network. The protocol is visualised in Figure 1(a). Simple and well known, it has become a popular benchmark for testing security protocol verification technology. We discuss it here because it is an important security protocol that nevertheless contained a significant error. This error was found through formal modelling and analysis.

Fig. 1.

Lowe [165] showed that, contrary to its intention, the protocol fails to ensure authentication. In particular, he demonstrated that an intruder can impersonate an agent A during a run of the protocol. The impersonator tricks another agent B into thinking that they are talking to A.

The protocol uses public key cryptography. Each agent A possesses a public key, which any other agent can get from a server. A also possesses a secret key that is the inverse of its public key. Any agent can encrypt a message using A’s public key, but only A can decrypt it, ensuring secrecy. The protocol also uses nonces: random numbers coined for single runs of the protocol.

Lowe encoded the protocol in CSP [123] and analysed it by use of CSP’s model checker, FDR [103]. Lowe did not direct FDR where to look in the protocol for a vulnerability—he simply modelled the intruder capabilities, but the exhaustive search carried out by the model checker found an attack in spite of this.

Suppose that I (an intruder) is a network user who can take part in network sessions. I can also intercept messages and inject new ones, but is not able to decrypt messages without the key. I can produce a new message in two circumstances: if I invents the nonce or if I already understands the message’s contents. This intruder can also replay complete encrypted messages, even without understanding the contents [213]. This approach is commonly known as the Dolev-Yao model [85].

The attack involves two simultaneous runs of the protocol, as shown in Figure 1(b). A establishes a valid session with I. At the same time, I impersonates A to establish a fake session with B. The flawed run of the protocol could be explained as follows: A sends a message with nonce \(N_A\) to I, who decrypts the message with I’s secret key. I relays the message to B, pretending to B that A is communicating. B sends \(N_B\) in response, encrypted for A, and so I relays this encrypted nonce to A. A decrypts \(N_B\) and confirms it to I, who learns it. I re-encrypts \(N_B\) and returns it to B, which convinces B that A is the other party. At the end of the attack, B falsely believes that A is the communication partner and that only A and B know \(N_A\) and \(N_B\) . This shows that the protocol is insecure. Protocol analysts call this a man-in-the-middle attack. Here, it has been discovered automatically.

Mondex. The Mondex application consists of smart cards with electronic purses (wallets) for use in the electronic commerce [235]. Customers use Mondex smart cards for low-value, cash-like transactions that need no third-party involvement. The Bank of England (the financial regulator in this instance) considered the requirements for Mondex to be security-critical: Mondex must have no implementation or design bugs that could allow electronic counterfeiting. So the developers certified Mondex to the highest standard available at the time. This was ITSEC Level E6 [129], equal to Common Criteria Evaluation Assurance Level 7 [57].¹ Mondex was the first commercial product to achieve ITSEC Level E6 (EAL7).

Reference [235] further describes the development of the Mondex application, with its abstract and concrete models. The abstract model describes the world of electronic purses: Atomic transactions exchange value and the abstract model expresses their required security properties. The concrete model is the purse design and the message protocol for value exchange.

The design team used the Z notation [233, 263] to specify both models. They proved that the concrete model is formally a refinement of the abstract one. This means that the concrete model respects all the abstract security requirements. The abstract model and its security properties is often easier to understand than the concrete model. Developers wrote manual proofs, believing that no efficient automated tools existed for such a large task. Instead, proof steps were type-checked using the fuzz ² and Formaliser tools [97]. Proofs were also checked by independent external evaluators.

There were four principal security properties:

•

The system and its users may not create value.

•

The system must account for all value.

•

Purses must have enough value for their intended transaction.

•

All transfers must be between authentic purses.

The design team changed a secondary protocol after the proof revealed a bug. A detailed account of the project is given in Reference [265]. Mondex has proved to be a dependably secure system, guaranteed by its formal development.

Tokeneer. The Tokeneer system was designed and developed by the US National Security Agency (NSA) [30]. ³ It provides secure access to an enclave of workstations with controlled physical entry. Access control requires biometric checks and security tokens. These tokens describe a user’s permitted actions within a particular visit to the enclave.

Developers needed to assure the security properties. They did this by conformance with the Common Criteria Evaluation Assurance Level 5 [57]. They also needed to show they could do this in a cost-effective way. NSA invited bids to use FM to develop a component of the Tokeneer system and then monitored this experiment to measure the effort and skills needed to perform the development.

Praxis (a UK company) won the contract and wrote a formal specification in Z [233, 263], formally refining the specification to a SPARK program. SPARK is a subset of Ada with an accompanying tool-set [29]. They proved key system properties and the absence of runtime errors, using traditional methods to develop extra software. These extra Ada programs provided interfaces with peripherals.

The project required 260 person-days, three people part-time, and nine months’ elapsed time. It produced about 10K lines of SPARK code with about 16.5K contracts. About 200 loc were written on average per day during the implementation phase, with about 40 loc through the entire project. A further 3.5K lines of standard Ada code were produced, with about 200 loc per day in the implementation phase or 90 loc throughout the project. System testing took about 4% of the project effort, much smaller than usual.

Two defects were found in Tokeneer. One was found using formal analysis, another was found by code inspection.⁴ The testing team discovered two in-scope failures: missing items in the user manual.

The task set by NSA was to conform to Common Criteria EAL5. The Tokeneer development actually exceeded EAL5 requirements in several areas: configuration control, fault management, and testing. Although the main body of the core development work was carried out to EAL5, the development areas covering specification, design, implementation, correspondence were accomplished to EAL6 and EAL7. Why? Because it was cheaper!

The seL4 Microkernel. The third-generation microkernel seL4 provides abstractions for virtual address spaces, threads, and inter-process communication. It provides an explicit memory management model and capabilities for authorisation. There is a guarantee that the binary code of the ARM version of the seL4 microkernel is a correct implementation.⁵ seL4 meets its abstract specification and does nothing else. In particular, the seL4 ARM binary meets the classic security properties of integrity and confidentiality.

The seL4 micro-kernel has a formal proof of its C code against its abstract specification [144]. This proof is machine-checked in Isabelle/HOL [194]. This assumes correctness of boot code, cache management, hardware, and hand-written assembly code.

The developers claim seL4 to be the only verified general-purpose operating system (OS) kernel. An operational model of the system forms as an abstract specification. A Haskell program prototypes the kernel. This prototype provides an automatic translation into Isabelle/HOL. The Isabelle code is then an executable, design-level specification of the kernel. This is hand-coded in C to form a high-performance C implementation of seL4. Refinement proofs link the specifications and the C code. Developers proved that attackers cannot subvert the kernel, not even if they use buggy encodings, spurious calls, or buffer overflow attacks.

1.3 Definitions/background

Here, we provide a set of common terms and definitions used throughout this article. This is particularly important, since the fields of formal verification and security developed independently for many years, and hence, some terms are overloaded and have slightly different meanings, depending on the context in which they are used. For example, in security (particularly cryptography), a certificate refers to a document that is used to bind an entity to a cryptographic key. However, within FM, certificate is used as a proof of correctness of a system or protocol.

Throughout this survey, on the security side, we use terms such as authentication to refer to the process of identifying and validating whether a user (an entity or individual) accessing a system is who the user claims to be. This is in contrast with authorisation, which is the process of allowing a user access to a system based on their identity. We furthermore often find security protocols designed to provide specific properties such as isolation, which is a design principle in which processes are separated and given privileged access to shared resources, e.g., shared memory (typically using techniques such as containerisation or virtualisation) or non-repudiation, ensuring that it is not possible to deny a validity of a statement or a message, especially in terms of its authorship. We would also like to point out the difference between anonymity where an identity of a user or a process shall not be disclosed and confidentiality where the content encoded as a block of data shall not be disclosed.

On the formal methods side it could be said that systems under consideration typically comprise a set of coordinated processes, which are program instances defining a set of instructions that are executed by one or more threads. We think of processes as being active entities in a system, as opposed to programs that are passive entities. Formal frameworks to describe the behaviour of processes include CSP, CCS, ACP, \(\pi\) -calculus, and so on. Processes typically implement protocols, i.e., a set of rules for transmission of data, and may synchronise over shared memory or communicate over a channel, which is an abstraction of a physical communication network. Shared memory implementations are increasingly complex due to the use of intermediate processor caches and may implement many different consistency models [6]. Similarly, one may place many different assumptions on a channel, e.g., FIFO ordering of messages; whether the channel guarantees integrity, availability, and confidentiality; whether the channel is error-free; whether message types can be distinguished, and so on.

In general, verification is with respect to a specification, which is an abstract (formal or informal) description of the allowable behaviours of an entity, e.g., hardware, a system, computer program, data structure, and so on. Formal verification often proceeds with respect to a model of a system, which provides a precise formal description of an entity, capturing the key characteristics of the entity being modelled. One must ensure that every feature described by a model is an actual feature of the entity. Different models of the same entity may be developed, depending on the properties that are of interest; a computer program for example may be modelled by relations between pre/post state; traces of states; functions between inputs and outputs, and so on. A model may describe behavioural functionality, protocols, and so on. In FM, one typically develops models at several levels of abstraction, with precise descriptions of the relationship between these levels. FM for security also requires a model of an attacker, e.g., the Dolev-Yao model, which is used in the context of communicating systems.

Within hardware verification, the term co-verification is used to prove that system software executes correctly on a representation of the underlying hardware design. It enables integration of software with hardware, before any physical devices (e.g., chips or boards) are available. The aim of verification is to ensure that it meets its implementation (of a specification), i.e., the physical manifestation of an entity. In some instances, one may refer to an implementation as a model that provides enough detail about an entity for the corresponding physical entity to be readily obtained.

To finalise the definitions it is important to revisit our classifications of FM. The first category we classify is theorem proving, where a proof of correctness of a system is provided in symbolic logic. This survey focuses on automated theorem proving, where proof assistants are utilised to generate the proofs. This method utilises a system of logic trying to determine if a statement \(\phi\) follows from a set of statements \(\Gamma =\lbrace \psi _{1},\ldots ,\psi _{n}\rbrace\) . The second class we consider is model checking where a finite state model of a system is utilised to systematically search the entire state space of this model (i.e., all possible system states contained in the model) for detection of counter-examples to statements about the system expressed as properties, for example in linear temporal logic as \(\square \diamond p\) stating that a property p holds in infinitely many states. As with theorem proving, we focus on automated model checking. Finally, we define a class of lightweight formal methods where we place methods that do not provide exhaustive analysis. A simple example could be static code analysis, where the code is analysed while not running to determine if it breaks predefined rules. Another example could be VDM [38], where the system is modelled in a formal modelling language and properties are expressed as contracts, i.e., pre, post conditions and system invariants, for example inv t == t.issueTime \(\lt\) t.expirationTime stating that universally t must be issued before it can expire. To perform the check, the model is animated and only specific scenarios are set (often those considered critical), however, VDM can also utilise combinatorial testing combining input paths to generate large amount of tests, hence significantly increasing the test coverage.

2 Survey

2.1 Categorisation and Overview

Since FM in security are applied across many domains, we structure the scope of the survey by presenting a categorisation based on a domain and a level of abstraction. This is done to provide a systematic overview of the wide field of FM in security. The labels for the four domains we have selected are:

Financial(Section 2.2): Aggregates the works applying FM in the area of finance/money as payment systems, home banking, financial markets, crypto-currencies. Examples are mobile banking apps, ATM infrastructures, the FIX stock exchange protocol, smart-cards/hardware wallets.

Industrial (Section 2.3): This label agglomerates works dealing with computing systems applied in the production of goods or services, manufacturing, and industrial control. Examples are a Water Treatment Management Panels, PLC control networks, Modbus/TCP, motor controllers.

Consumer (Section 2.4): It categorises works focusing on the security of end-users’/ individuals’ personal computation devices and applications such as a command-line shell, a home operating system, a Voice over IP protocol, and an exercise smart appliance.

Enterprise (Section 2.5): This is the dual of the Consumer category, as it is used to group the works focusing on the security of corporate systems providing computing services satisfying the needs of organisations instead of individuals. Examples are email services, e-government systems, the sn2 protocol, data server warehouses.

As presenting the four domains would only separate the FM in security research by the field of application, we further present five levels of abstraction at which the formal verification is carried out. These levels of abstraction are:

Application:

Used for works that apply FM for security at the application or purpose of computation level.

System:

Used for works that apply FM for security within the architectural level, often encompassing multiple subsystems.

Protocols:

It is used to apply FM to assert properties or analyse communication protocols between system components level.

Implementation:

This is a cross-cutting category encompassing all the works that focus on application/usage of FM directly on the resulting system (e.g., runtime monitoring) instead of emphasis on designs and specifications.

Hardware:

Used to classify works applying FM in the process of hardware development.

This categorisation allows us to systematically review the state-of-the-art-research and provide an overview based on this. To provide a clear overview of FM in security, we further apply a third dimension, defining the type of the FM used, i.e., model checking, theorem proving, and lightweight FM. This provides a quantitative overview of different research works within Figure 2.

Fig. 2.

The sections within this survey follow a logical organisation, where the research works are grouped together by the type of the application, system, protocol, implementation, or hardware that they are applied to. Within this grouping, the research works are further organised into paragraphs following logical categorisation. As an example, the first paragraph could consider works related to manufacturing, while the next paragraph considers works related to industrial control. In both cases the research is aimed at industrial domain but with a different scope. Each section within the survey represents a single domain, where we present a systematic summary of research works belonging to different levels of abstraction. As this survey attempts to categorise large amount of research works, each of them is only presented briefly focusing on the analysed problem and utilised FM technology. For more details on individual research works, we point the reader to a technical report that this survey is based on [149].

2.2 Financial

Financial computing, including banking systems, independent budgeting applications, and mobile payment applications, is a rapidly developing field. This section provides an overview of how FM have been used to analyse the security of banking mobile applications, alternative currencies, such as cryptocurrencies, smart contracts, banking backend systems, electronic trading systems, payment protocols, cryptocurrency hardware, and wallets. On a different level of abstraction, it could be seen within this section that the system level consists of most research works. This section also mentions the legal challenges when applying FM to financial systems. These legal challenges arise from limited access to these systems and a certain level of avoidance of publication of potential vulnerabilities by vendors of these systems, i.e., often making it difficult for researchers to get deep insight into these systems.

The method most used to analyse security within the financial domain is model checking, and authors apply different model checkers to specific problems. This could be attributed to the fact that the different entities whose security is being analysed lend themselves well to be modelled in a state transition representation and also that their state space is sufficiently limited to be analysed without issues such as the state space explosion. The cyber security topics present within the financial section are shown in Figure 3.

Fig. 3.

Application. Nowadays, banks not only provide mobile applications, but whole alternative currencies are being developed. This rapid growth provides many opportunities for use of FM on an application level.

An application of FM to banking apps could be found in Reference [63], where the authors analyze security of apps from 15 leading UK banks discovering several vulnerabilities. The authors proposed a correction to one of the flaws, which was formally verified using ProVerif.

Another widespread security threat in banking is malware. In this area, the authors of Reference [152] analysed Android banking applications using Krakatau byte-code tool⁶ to generate the Java byte-code of the application, further translating it to Calculus of Communicating Systems (CCS) [181]. The authors then dispatched the CCS model to the Concurrency Workbench of New Century (CWB-NC) model checker [69] searching for malware properties. The authors have accomplished 98% malware detection rate. Similar approach was utilised by the authors of Reference [126] focusing on the banking SMS messages.

In the alternative currency area, the authors of Reference [249] analyse the Electrum Bitcoin wallet by creating a model in ASLan++ [253] of the two-factor authentication utilised by the wallet. The authors have uncovered potential vulnerabilities using the Cl-Atse protocol analyser [248].

Smart or programmatic contracts are also an important aspect of modern financial landscape. The automated contract enforcement requires an implementation that shall be free of vulnerabilities. This led to use of FM for creation of certified contract languages [17] and certified virtual machine byte-code [199].

A specific survey has been carried out in this area, providing more detail on utilisation of FM [180].

System. The financial systems of today could be categorised as classic systems such as SWIFT inter-bank network system and new systems often introduced by smaller players or by regulatory pressures [259] that tend to push the sector to move faster.

Most of the FM works related to classic systems are decades old with few works such as use of BAN logic for analysis of mobile payment system [7] or use of SPIN tool for analysis of ATM systems [197] or internet payment systems [272]. In Reference [216] the authors show legal barriers faced by proponents of FM in the domain.

Within the area of Electronic Trading Systems (ETSs) there is a trend of application of FM to decentralised systems such as blockchain-based cryptocurrencies. These works range from verifying algorithms in a cryptocurrency platform [268], analysis of Ethereum smart contracts [121], to verification of the blockchain system as a whole [87]. Other works have considered building models of Europay Master Visa standard by use of automata learning techniques [2] or use of lightweight formal specification in fraud detection [209].

Protocols. Protecting financial transactions is a work well suited for FM. Specifically in the case of Near Field Communication (NFC), a short-range radio communication technology, utilised in contactless payments. This method has several vulnerabilities [175], where the authors of Reference [168] propose and verify an NFC protocol suing the Scyther tool [73] addressing several vulnerabilities. In similar fashion, in Reference [5] the authors propose a protocol for securing of NFC payments and analyse it using the FDR model checker. Authors of Reference [11] have focused on analysis of security of NFC enabled forgery protection also utilising the FDR model checker, discovering several potential attacks and providing mitigating measures. In the field of authentication protocols the authors of Reference [266] verified the mutual authentication properties of a secure electronic protocol using SPIN tool, discovering several vulnerabilities. Similarly in Reference [118] the authors have analysed a biometric transaction authentication protocol using ProVerif [41] and proposed fixes to discovered vulnerabilities.

Finally, the authors of Reference [44] have proposed a secure SMS based protocol for mobile payments and analysed it against several security properties using AVISPA [21].

Implementation. Since vulnerabilities in financial software could lead to financial losses, use of FM can provide a substantial benefit in this area. Smartphone applications are often used as a gateway to financial services. The authors of Reference [127] utilised static analysis tools, discovering that financial applications from developed countries contain less vulnerabilities than those from developing countries. Similarly, the authors of Reference [241] have statically analysed over 10,000 Android applications to compare the security of financial applications with the rest, which led to a discovery of a worrisome trend where the analysed applications have gained more vulnerabilities within a span of two years.

The authors of Reference [98] have modelled 80% of EMV2, a successor to EMV, in VDM to provide a formal model for the implementation and analyse security attributes of EMV2. The authors have further attempted to code generate parts of EMV2 to Java directly from the VDM model.

Another aspect of financial software is use of open APIs. The authors of Reference [94] have modelled a financial grade OpenID API as a set of theorems, discovering several vulnerabilities and proposing fixes to these. Finally, the authors of Reference [16] have analysed bitcoin contracts using UPPAAL, determining the secure time to live within the contract protocol. Similarly, other types of smart contracts are being utilised [35]. The authors of Reference [199] have verified Ethereum smart contracts such as the ERC20 token contract [90] using K-framework’s reachability logic theorem prover [234], discovering that the token implementation that diverges form the ERC20 specification contains vulnerabilities.

Hardware. In the area of contactless payments, the NFC hardware can pose security challenges. To address one of these challenges, the authors of Reference [62] have introduced a scheme to prevent relay attacks based on a distance bounding protocol [46] and verify this scheme using ProVerif.

Cryptocurrency is often associated with a specific hardware, where in several cases FM were used for security improvement. To this end, the authors of Reference [22] have proposed a device for approval of security critical operations. The authors have verified a property of deterministic start of the device using an SMT solver. Similarly, the authors of Reference [171] have utilised theorem proving to check an unforgeability property of a hardware wallet to answer a question: what if the manufacturer of the wallet cannot be trusted? Finally, the authors of Reference [19] have also attempted to prove security properties of several hardware wallets using theorem proving, showing that the wallets were secure only under specific assumptions.

2.3 Industrial

Industrial processes are a backbone of a modern society, as they provide control not only for production of necessary goods, but also utilities such as electricity and water treatment. This section provides an overview of how FM have been utilised in security analysis of automotive control applications, robotic applications, PLC software, industrial communication protocols such as Modbus and OPC UA, SCADA systems, and hardware devices underpinning industrial computing. An interesting note is that the research works are distributed uniformly across the different levels of abstraction, demonstrating that all aspects of industrial computing have been scrutinised using FM to provide either security analysis or security assurances. In the industrial application of FM the problem is often considered domain-specific, i.e., cyber security properties are based on whether the considered industrial system is, for example, an automotive controller or a water treatment plant.

As within the financial section, the most-used FM to analyse the security properties is model checking. This could once again be attributed to the nature of the problem where, for example, PLC programs and industrial processes lend themselves to be easily modelled using state transition systems. As some of the industrial computing is complex, the problems are often modelled more abstractly to avoid the state space explosion problem. Within the hardware level of abstraction, however, in industrial computing, theorem proving is often the FM of choice, as it allows description of the hardware in more detail. The cyber security topics within the industrial section are shown in Figure 4.

Fig. 4.

Application . Industrial applications are often used to control critical processes, hence FM could provide strong assurance of security. The authors of Reference [120] utilised model checking based on automated translation of automotive ECU applications to CSP [122] and subsequently dispatching the model to the FDR model checker. Discovered counter examples were then provided to the implementation team.

Industry of the future utilises interconnectivity and robotics. In Reference [193] the authors have analysed a security of an application controlling a cap attaching robot by creation of a model using Maude [179] and two attacker models, discovering possible attack vectors. Similarly, the authors of Reference [254] analysed security of applications based on the Robotic Operating System [205], expressing the security properties in CTL and utilising the UPPAAL model checker [159]. The authors then automatically generate the implementation C++ code.

The authors of Reference [177] have focused on security analysis of a water treatment SCADA system by use of the system logs. The application behaviour was modelled as timed automata, while the security properties have been expressed in timed temporal logic [12], providing 100% attack detection rate based on log data. The authors of Reference [273] have successfully utilised the Z3SMT solver [80] for PLC malware detection, while the authors of Reference [146] used the NuSMV model checker [65] for automated detection of intrusion code, demonstrating the usefulness of FM in this area.

System . Cyber attacks against industrial systems could have severe consequences [258]. To mitigate this, the authors of Reference [211] have created a formal model in ASLan++ of a real-world industrial control system and carried out attacks utilising a Cyber Physical Dolev-Yao attacker [210]. The analysis with help of CL-Atse analyser has discovered seven out of eight possible attacks. Similarly, the authors of Reference [82] mitigate the cyber attacks by proposing a formally verified security framework for industrial control systems. The verification was carried out using ProVerif and proved the security aspects of the system utilising the framework. Furthermore, the authors of Reference [114] verified a PLC program using timed automata and UPPAAL ensuring that the program was not compromised. In a similar fashion the authors of Reference [255] have modelled a water-level control system in timed automata utilising the PAT model checker [237] to successfully verify security recovery mechanisms of the system.

Within the area of connectivity the authors of Reference [214] have demonstrated formalisation and analysis of firewall rules using the Z3 SMT solver, lowering the errors in firewall configurations. In Reference [150] the authors have utilised TLA+ [158] to ensure effectiveness of mitigations strategies against several cyber attacks, while the authors of Reference [246] have used combinatorial testing within VDMJ [161] to generate 145 million tests for a formal model expressed in VDM-SL [160] of an industrial control system, providing assurance for several security properties.

Protocols. Industrial communication protocols carry critical data, requiring a high level of security. The authors of Reference [224] have formally analysed security of the Modbus/TCP using Coloured Petri Nets [138] combined with Formal Component Analysis [201] discovering a possible attack. Similarly, the authors of Reference [187] modelled the Modbus protocol as a Dynamic State Machine [186], automatically translating it to Promela for verification using the SPIN model checker. The authors have uncovered a possible man-in-the-middle attack. Another protocol, OPC UA, has been analysed by the authors of Reference [203] using ProVerif finding vulnerabilities in the authentication sub-protocol. In similar fashion, the authors of Reference [86] analysed several Modbus and OPC UA authenticity and integrity properties using TAMARIN theorem prover [176], discovering the necessity for secure channels. In Reference [15] the authors have formally analysed the authentication properties of DNP3 protocol utilising the CPN state space analysis tool [137], discovering a potential for replay attack. Finally, the authors of Reference [49] have analysed an authenticated CAN protocol using ProVerif, discovering that limited use of cryptography allows for a replay attack with partially modified data.

Implementation. Since industrial software often consists of a large codebase it could be difficult to be formally analysed in its entirety. Over the years, several approaches to code analysis have been presented; for example, the authors of Reference [18] use a dialect of UML, SysML-Sec to model of a large industrial codebase, iteratively refining the model and automatically translating it to \(\pi\) -calculus using the TTool [88] and analysing it using ProVerif. The authors propose for this approach to be integrated to the software development process.

In the aeronautics industry the authors of Reference [70] have created a formally verified implementation of unmanned aerial vehicle using several tools; for example, the jKind model checker [99], satisfying correctness of components and Isabelle/HOL theorem prover [194] to assure that system execution semantics matches the model. This approach has successfully prevented cyber attacks against the vehicle.

As PLCs are the backbone of industrial automation, a lot of focus have been put into ensuring that the PLC programs satisfy security properties. For example, in Reference [223] the authors use state transition diagrams of PLC programs as a basis for formal model dispatched to NuSMV model checker, while expressing the security properties in LTL. Similarly, the authors of Reference [247] utilised Petri Nets to develop software falsification detection by translating Petri Nets to Promela and dispatching it to the SPIN model checker, while modelling the falsification properties in LTL.

Hardware . FM for hardware verification is a well established field with specification languages such as Verilog [243] and VHDL [188]. The authors of Reference [124] focus on co-verification of Intellectual Property Blocks (IPs) for use within System on a Chip (SoC) architectures, considering technologies such as secure boot and concurrency in a time-of-check-to-time-of-use considerations [48]. The authors utilised semi-automatic co-verification methodology using a toolchain comprised of Boogie [31] as intermediate verification language, through Corral software verifier [157] and SMACK [206] for bit-precise checking with an ultimate goal of producing secure SoCs. The authors of Reference [162] consider that all software layers could be compromised and have developed an application-specific hardware monitor based on a formally analysed C code and a junction box validated in a hardware description language with a goal to monitor the hardware controller for malicious activity. The authors model their hardware monitor in Frama-C [74] with Jessie plugin, allowing for automatic deductive verification using Why [95].

Within the area of integrated circuits, the authors of Reference [164] consider the trustworthiness of hardware using Proof Carrying Code (PCC) [189], utilising Coq to derive theorems for the hardware descriptions annotated with PCC. This has been later extended in Reference [110] to a notion of a Proof Carrying Hardware (PCH), utilised to verify security of IPs supplied by untrusted vendors by extending the VHDL and utilising Coq to carry out the verification of security theorems. Finally, the authors of Reference [3] consider the possibility of hardware trojans being injected during manufacturing and utilise the nuXmv model checker [56], while specifying the hardware properties in LTL to detect these trojans.

2.4 Consumer

Consumer computing such as use of personal computer, smartphones, and underlying connected services is an integral part of modern life. Consumer computing has often been characterised as of less critical nature than for example industrial systems, however, this view is changing as society introduces more digital technologies to everyday life. This section provides an overview of use of FM in analysis of cyber security of consumer computing, ranging from consumer electronics for fitness equipment, mobile operating systems, web browsers, consumer Internet of Things (IoT) devices to commodity hardware for devices such as personal electricity meters. An interesting fact within the consumer domain is significant use of so called lightweight FM, utilised often not only on the application level of abstraction, but also considering implementation and hardware. One challenge in formal analysis of consumer systems is a rapid nature of evolution of these systems, where the competition in consumer markets often forces fast adoption of new technologies.

Once again the most utilised FM is model checking. It could be argued that this is due to the significant model checking experience gained in other domains. Many consumer computing entities are, however, complex, interconnected chains of services, which could explain wider utilisation of lightweight FM, especially as some of these are used directly to build chains used to construct the consumer computing entities. It should also, however, be stated that theorem proving has also been utilised on all the different levels of abstraction within the consumer computing domain. The cyber security topics present within the consumer section are shown in Figure 5.

Fig. 5.

Application . FM thrive in checking consumer applications for malware. A practical definition of malware (one that can be used to classify executable files) intersects the perspective shift FM advocate: focus on “what” is computed instead of the “how.” Quoting Reference [147]: “any (formal) definition of the concept of malware depends on the definition of the concept of software system correctness.” Also, a majority of malware is the product of tools generating variants of known vulnerabilities/attacks or known malware. The authors of Reference [66] show variants are easy to hide syntactically, but not semantically.

Model checking-based approaches provide malicious behaviour semantic signatures by providing counterexamples. Recent approaches as in References [229, 231] extract push-down automata as models. A promising area is the application of the techniques to the realm of the Android operating system [230]. Although successful, the FM techniques provide no panacea to consumer malware protection. The malware game advances with discovery of zero-day (latent) vulnerabilities. FM have been argued to avoid these vulnerabilities in the first place [174], but practically, new malicious behaviours are expected to appear, thus the problem becomes to learn malicious behaviours. There are several proofs of concept where FM leverage the signature learning either in terms of or using push-down automata reachability in the process [75, 76, 167]. There are a few works where theorem proving is applied in malware [227], but the number of publications is small, and it is difficult to ascertain if there is an effective gain from it. Perhaps model checking is more appropriate to the domain due to its non-interactive nature, since malware is inherently a game between attackers and an algorithm.

System . Recently, we have seen the adoption of the software marketplace paradigm to prevent attacks and malware to reach consumer systems. For instance, Android enforces permissions at the application level, which could, however, lead to privilege escalation [78]. The authors of Reference [26] propose a tool-based approach, called COVERT, for compositional analysis of Android inter-app permission leakage. COVERT assesses the security of a system as a whole by generating Alloy specifications [131] and analysed more than 500 real-world applications, confirming the findings previously found within References [23, 91], showing that many Android applications are over-privileged. The authors of Reference [25] have moved towards analysing the permission protocol itself, identifying design flaws where two applications can apply the same custom permission, resulting in first installed application being able to access the resources of the second.

Operating systems usually contain an underlying security model amenable to FM checking. In Reference [83] the authors have verified a proposed access and integrity control for a Linux-like OS, using Alloy and Event-B [4]. While the authors have experienced scalability issues with Alloy, the analysis has uncovered bugs that could become more serious if discovered in the implementation phase. Another example is Reference [170], where the authors verify security policies in the form of invariants annotating the code of ExpressOS, a secure OS alternative to Android. The authors utilise automated theorem provers and report the verification overhead (added annotations) was roughly 2.8% source code.

Consumer IoT systems such as smart home devices pose security vulnerabilities and have been widely verified using FM. The authors of Reference [153] use model checking tools within AVISPA and by BAN logic to verify a framework ensuring anonymity, authentication, and integrity in smart home environments. In Reference [182] the authors have developed the IoTRiskAnalyzer tool used to help engineers apply the most fitting security policies. This has been achieved using a Markov Decision Process [202], formalisation of risk properties as probabilistic CTL formulas, and verification using the PRISM model checker [156]. Car manufacturers are also taking advantage of connected devices, especially smartphones. For instance, the authors of Reference [53] have developed a smartphone-based immobiliser with formally verified protocol using ProVerif against a Dolev-Yao attacker model to ensure strong guarantees of security requirements.

Protocols . The area of consumer communication protocols covers text and multimedia communication lending itself to formal verification of security.

In Reference [71] the authors have created a formal model of the Signal protocol in terms of predicates and theorems and have applied theorem proving, resulting in improvements in the use of the protocol’s random generator.

Security of the consumer communication systems often depends on the mechanisms introduced in the Needham-Schroeder [190] authentication protocol and the Denning Sacco protocol [81] for secret key distribution. The authors of Reference [60] have created a simplified model of the Needham-Schroeder NPSK protocol and the Denning Sacco protocol and expressed security properties using LTL. The authors have provided an efficient model for model checking of the security properties using the Spin model checker.

Implementation . The recent adoption of FM tools by large technology companies has shaken up the field. If in the past FM were tied to niche safety critical domains (e.g., aerospace, railway, medical) and fields with significant governmental regulation, the current panorama shows that the future brings the usage of FM tools in the daily practice of software engineering. No matter the intention behind the usage of FM tools, the outcome has demonstrated a contribution to increasingly secure implementations. According to recent reports, when a developer commits a code modification to one of the large technology companies’ codebases, a static analysis tool is invoked and a code review is provided. The author of Reference [196] describes the process as continuous reasoning, and any change to a Facebook product is analysed by the Infer static analysis tool, which checks “small theorems” on large codebases. This approach has been shown to improve the security of the company’s own codebase and library implementations (e.g., OpenSSL). The same is reported about the software engineering practice inside Google [215], although it is not clear whether FM is used by Project Zero, its elite security team. However, the authors of Reference [24] report on how numerous security vulnerabilities were fixed by applying FUDGE, a static analysis tool based on fuzzing developed in house.

Particularly targeted to the security domain, the authors of Reference [84] report a static analysis tool, Zoncolan, in collaboration with the Facebook App Security team. Zoncolan uses abstract interpretation to analyse and issue security alerts for the implementations of the applications in the company’s codebase: Messenger, WhatsApp, Instagram, or Facebook. This level of application of FM shows the implementations of software used by millions of consumers has been swept by a FM tool.

FM is also being applied to secure implementations of web browsers, which are designed with security in mind because they mediate a vast amount of personal information (e.g., credentials, banking details). Nevertheless, due to a large attack surface of a web browser, attacks are possible, and implementation flaws are not uncommon. In a bolder move, the authors of Reference [133] propose a new browser, QUARK, that follows the “kernel architecture”⁷ of modern browsers, but QUARK’s kernel is formally verified. The formal verification yields to the Coq theorems to assert properties as tab non-interference, or cookie confidentiality and integrity. According to work in Reference [96], the price to pay for such a prime example of functional correctness verification (above airline runtime error-free level) is 25% increase in overhead, affecting performance. Increasing browsers’ security risks, browser extensions can spy and exploit users as demonstrated in Reference [109]. In Reference [218] the authors report on a verified design of an experimental browser using the Maude tool and rewriting logic, and the authors of Reference [184] show that x86 native code executed by arbitrary clients conforms with a predefined sandbox policy when using Google Chrome’s Native Client service.

Hardware . In contrast to critical-system hardware (e.g., fly-by-wire hardware) attackers cannot be prevented from physical access to the consumer hardware, which provides a large attack surface. Modern consumer hardware provides hardware-level protections for critical software components. An example of this is ARM TrustZone [192], providing separation between trusted and rich software providing potentially untrusted interfaces. The authors of Reference [93] propose verification of hardware security properties by use of information flow control at the level of the Hardware Description Language (HDL) such as SecVeriLog [271]. The authors create SecVeriLogBL, an extension to SecVeriLog, by adding new types for security labels defined in SecVeriLog. This allows for static analysis at the design time, providing a lightweight verification with small effects on the hardware performance. To demonstrate this approach, the authors have designed an implementation of TrustZone, including 10+ security bugs. Similarly the authors of Reference [163] present a formally defined hardware security enforcement for x86 architecture. In this setting, the software relies on underlying hardware for security enforcement; for example, memory paging features of an x86 CPU. The authors note that incorrect implementations of hardware enforcement policies often lead to vulnerabilities [140]. The authors use Coq to model the architecture and the Coq theorem prover to prove the soundness of the security policy.

In the area of commodity hardware, the authors of Reference [239] have used model checking to determine possible attacks on smart meters, which are considered critical devices [143]. The authors have created a model of the smart meter using rewriting logic, formal definition of the attacker’s actions, and used the Maude tool to check that the attacker’s actions are not able to break the security invariants. The discovered attacks were then mapped to an implementation of a smart meter, the SEGMeter, to investigate the practicality of these attacks. The authors determined that many attacks discovered by the model checker are indeed practical, despite the model being abstract and not specifically refined towards the SEGMeter implementation.

Today, hardware is often packaged as an SoC, which security may be verified using the combination of integrated theorem proving and model checking proposed in Reference [111]. Due to the hierarchical nature of SoCs, the authors propose that the design expressed in HDL is decomposed into sub-modules and security specifications into sub-specifications. The sub-specifications are then verified using the Cadence IFV model checker [1]. These verified sub-specifications are then used as proven lemmas in the Coq theorem prover [242], removing the need to prove these lemmas by hand. This simplifies the model checking as well by providing only a small specification to the model checker, avoiding the state space explosion. The authors extend their method by automated code conversion from HDL to verifiable specification [112]. SoC complexity increases in Multi Processor SoCs (MPSoC), where multiple processors exchange data via Network on a Chip (NoC) routers. The authors of Reference [220] have used unbounded model checking to verify security properties of an NoC, which was practical due to the highly sequential behaviour of NoCs. The authors formalise the security and functionality correctness properties using LTL and use the CIP unbounded model checker [155] to verify them. As a proof of concept, the authors have analysed six different router implementations, determining the feasibility of their approach for NoC security analysis in early design stages.

2.5 Enterprise

Enterprise and large corporate computing is the backbone of large international business. In recent years, there is a trend in enterprise computing to utilise cloud solutions, while still often operating on-premises (local) data centers. These data centers and cloud clusters are utilised for a plethora of enterprise tasks such as virtualisation of collaboration platforms, company management, and hosting of corporate web portals. This section provides an overview of utilisation of FM to address security challenges of enterprise computing, ranging from secure data storage through virtualisation and software-defined networking security to strong authentication using hardware tokens. As enterprises are larger entities, changes are often slower and need to be well managed. To this end, the FM have been utilised as a booster in cloud adoption by enterprises, as several FM-based solutions have been proposed to enable enterprises’ secure switch from local data centers to federated cloud solutions.

Similarly to previous sections, model checking is the most used tool in formal analysis of security in enterprise computing. Theorem proving is, however, not far behind, especially within analysis of hardware such as Trusted Platform Module chips within enterprise servers. Lightweight FM have also been significantly utilised at the implementation level of abstraction, since they are often provided as plugins to software development environments, making them easily accessible. The cyber security topics present within the enterprise section are shown in Figure 6.

Fig. 6.

Application. Enterprise applications often process and store data critical for an organisation. Nowadays, such data is carried by Software-Defined Networking (SDN). The authors of Reference [226] have created a verification platform for applications utilising SDN, consisting of a modelling language that could be automatically translated and dispatched to PRISM, SPIN, and Alloy model checkers. Any counterexamples are then displayed in the tool. Similarly to SDN, Service-Oriented Architectures (SOAs) are often used within enterprise applications, increasing the interconnectivity of these applications. In Reference [20] the authors present a platform for security assessment of SOAs utilising a formal specification language and several model checkers, namely, CL-Atse and OFMC. The authors uncovered an issue with SAML-SSO integration in Google Apps.

Enterprise data are often stored within large relational databases. In this context, the authors of Reference [61] have proposed a method for secure outsourcing of databases to untrusted servers, building upon the notion of Verifiable Databases [36], and utilised theorem proving to demonstrate their method as secure. Another often used technology in the cloud is virtualisation. The authors of Reference [222] introduce a formal analysis scheme for security of Xen hypervisor, consisting of model checking and static analysis, successfully rediscovering a known vulnerability. Finally, we would like to point to the work within Reference [72] describing how a leading cloud provider utilises FM for security of their services, noting that the benefits of FM are important to their customers.

System. Nowadays, large enterprises can either host their own infrastructure, fully utilise the cloud, or partially combine their infrastructure with the cloud, leading to federated cloud systems [183]. The authors of Reference [270] have proposed a method for analysis of federated cloud behaviour utilising CPN and CPN tools [137], creating several models for security analysis. Similarly, the authors of Reference [256] have used Z with Z/EVES theorem prover to formally analyse a data exchange system against confidentiality and integrity properties, while also generating tests utilising the domain theory [39]. In Reference [135], the authors present a formal approach to analysis of firewall rules and cloud topology based on Mobile Ambients [55] and the non-interfering Boxed Ambients calculus [51]. As cloud computing is often built utilising shared resources, the authors of Reference [169] have built an offline framework for formal analysis of network isolation properties, ensuring isolation among shared resources. This has been carried out by use of first order logic and the constraint satisfaction solver Sugar [240]. Similarly, the authors of Reference [173] have proposed a security framework for cloud complexity management (agent) system [113], utilising the Z/EVES theorem prover to analyse several cloud security properties within a NIST [43] cloud reference architecture. Also, in Reference [232], the authors have proposed a broker solution for automatically pairing cloud services with customers while managing the cloud complexity. An important part of the broker is finding a service satisfying customers security requirements, defined in first-order relational logic [130] using KODKOD finite model finder [244].

Several works also consider virtualisation within the cloud system. For example, the authors of Reference [117] have proposed a formal framework for analysis of security and trust in virtualised system, combining hardware and software models expressed in CSP# [238] dispatched to the PAT model checker discovering a subtle bug in a real-world cloud system. Similarly, the authors of Reference [42] have proposed a security subsystem for change analysis within virtualised infrastructures in relation to security policies by utilisation of graphs and graph transformations dispatched to GROOVE model checker [102].

Protocols. Enterprise computing is moving towards the cloud, creating need for secure communication protocols, benefiting from FM analysis.

Amazon cloud services [13] use the s2n [14], the open source implementation of the TLS protocol, utilising FM to prove its correctness. For example, in Reference [64] the authors demonstrate that the Hashed based Message Authentication Code (HMAC) utilised by the protocol is indistinguishable from a random generator using Cryptol specification language [89] describing the HMAC, which was then dispatched to Coq theorem prover and the results are connected with the implementation by use of Software Analysis Workbench [100]. In the Microsoft cloud, the authors of Reference [136] have developed a tool for analysis of network protocols to assist with the task of network policy maintenance within data centers by use of the Z3 SMT solver, providing an important security tooling for Azure cloud services. Since the cloud services are often accessed remotely, the authors of Reference [151] have utilised Alloy analyser to find vulnerabilities in the SAML protocol [125].

Nowadays, clouds collect data from small-footprint IoT devices [185], which prompted the authors of Reference [141] to propose a lightweight mutual authentication protocol and verify it against several attacks using OFMC and CL-AtSe. Similarly, the authors of Reference [212] have proposed a mobile authentication scheme verified using ProVerif. The IoT devices could take advantage of the 5G networks, where in Reference [33] the authors use Tamarin, finding an issue with the authentication sub-protocol, while the authors of Reference [8] have analysed the authentication framework protocol [269] and the mobile ethernet protocol [128] by expressing them in the CSP, which was subsequently dispatched to the FDR model checker for analysis against mutual authentication properties.

Implementation . Enterprise computing is often composed of many applications implemented using different technologies. For example, the authors of Reference [195] have created a static code analysis tool for PHP plugins, phpSafe, that was then utilised to discover over 580 vulnerabilities in several PHP plugins. In similar fashion, the authors of Reference [267] have created a tool utilising invariant analysis [116] for malicious behaviour detection, noting the high effectiveness of logic flaws in several web applications.

Hypervisors are an important part of enterprise computing. To the end of security implementation of hypervisors, the authors of Reference [252] have created a framework for implementation of security verified hypervisors based on behavioural contracts and verified using FRAMA-C [225] for static analysis of the behavioural contracts. Similarly, the authors of Reference [251] have created a hypervisor framework for verification of memory integrity within single guest hypervisors utilising the CBMC model checker [67] for automated analysis of most of the codebase.

Hardware. Enterprise computing requires significant cloud hardware infrastructure and assurances such as data confidentiality and computational security. Customers often consider a cloud provider as an untrusted entity, where the administrators themselves could pose a security threat [217]. In this regard, the authors of Reference [219] created a cloud isolation system, separating the user data from administrators and limiting the operations that the administrators could take against a user’s virtual machine, utilising a hardware module, that the authors named Trusted Cloud Module (TCM), which provides a limited set of interfaces to the cloud administrator, manages encryption keys, and provides secure storage for the user. The module is built from off-the-shelf hardware components using the Scyther verification tool. Similarly, the basis of any trusted computing is the Trusted Platform Module (TPM) co-processor providing secure storage and computing environment. Unfortunately, the security of platforms using TPM is often not formally verified leading to vulnerabilities [50]. To mitigate this, the authors of Reference [27] have proposed TRUSTFOUND, a formal modelling framework for model checking utilising a Trusted CSP#, an extension of CSP#, and LS \(^2\) [77], where the PAT model checker is used for verification and detecting six implied assumptions and two severe logic flaws.

Sometimes, to provide strong authentication, small One Time Password (OTP) generation hardware is used by enterprises to authenticate users towards cloud services [45]. One such device is Yubikey, a USB OTP generator. In Reference [154], the authors have formally analysed the security of the Yubikey OTP and also a security of the Hardware Security Module (HSM) . Another challenge in this area is addressing CPU side-channel attacks. One of these attacks is a timing channel attack, where an attacker, possibly a virtual machine, could determine the algorithm executed by another virtual machine in a shared environment. To solve this, the authors of Reference [92] have proposed Timing Compartments, an isolation scheme implemented in hardware isolating timing information between parties sharing the resources. The scheme was checked by information flow analysis using SecVerilog.

3 Future Outlook

This survey has provided an overview of use of FM within security in several domains. Based on the research conducted within these domains, it is expected that in some cases the use of FM will accelerate while, in other cases, the use will increase with a slower pace. There is, however, a general trend of increase of adoption. In cases within the financial domain, it is clear that the use of FM comes with new financial technologies such as cryptocurrencies and smart contracts. This adoption could be seen in a survey aimed specifically at the smart contracts domain [119]. The use of mobile applications in the financial domain is also spurring a demand for high security assurance that could be delivered by use of FM. Finally, with the rise of cryptocurrencies, the hardware within the financial domain is being specialised to facilitate transactions. It is expected that the security of this hardware will continue to be scrutinised formally with increasing coverage and complexity.

The industrial domain faces its own set of unique challenges in the area of cyber security. It is expected that with increase in automation complexity and use of digital technologies in critical industrial installations, FM will play a crucial role. The trend was already presented in 2015 by Reference [148], who have surveyed approaches for security and safety of industrial control systems, including informal approaches. As of now, most works within this domain are of reactive nature, i.e., analysis of existing systems and protocols. However, several works within the survey show a trend towards utilisation of FM early in the design process of new industrial installations and protocols. Another emerging trend within the industrial domain is integration of formal verification tools with the software development processes, this is clear primarily in terms of robotic applications and PLC code. It is expected that with increasing complexity of robotic applications and underlying hardware, FM will play a significant role in the future.

The domain of consumer computation is a rapidly evolving one. The consumer trends move fast; however, a somewhat surprising amount of work is already put forward to use of FM in malware protection [37], going as far as creation of formally verified Internet browsers. Also, the shift of computation from computers to smartphones has brought new security challenges. In this area, a lot of focus has already been put on analysis of the Android OS permission system. This area shows an increase in the amount of research and as long as the mobile OSs are in use by millions of users, the formal verification will accelerate, possibly leading to full formal security verification of a popular mobile OS. Consumer hardware such as smartphones is putting into use TPMs, securing mobile computation. Also in this area, the use of FM is increasing, specifically to ensure the properties of the secure TPM enclave.

Finally, the domain of enterprise computation has uncovered several interesting trends. The first of these is a trend to use FM against virtualisation hypervisors to analyse security properties of existing virtual environments as well as use the knowledge to build fully formally verified hypervisors. Another important trend is the significant investment that major cloud computation providers are putting into formal verification of security of their products. To this end, not only have existing tools been applied, but the cloud providers have turned towards development of their own FM tools. Both of these trends are expected not only to continue but also to accelerate due to the ever-increasing popularity of cloud computing and virtualisation. Several of these trends were already mentioned in 2002 by Reference [139].

It is important to note that several authors have expressed a wish for improvement of automated formal verification tools. This is to allow for simplified entry of non-practitioners to the world of FM. Both academia and industry are moving towards addressing this wish, with tools becoming similar to software development IDEs and in some cases integration of FM toolkit directly to an existing IDE. It is expected that knowledge of FM will become important for system and software engineering disciplines in the future, and therefore collaborative projects between industry and academia shall provide experts in this domain. These issues have been discussed for several years now [79].

When it comes to FM techniques, static analysis tools are becoming popular in software development, while model checking is moving strongly towards system design and protocol verification, with model checking being designed for specific problems. Theorem proving is also showing a promise of playing a crucial role in the future, given that the perceived large learning curve could be minimised.

As the use of FM is accelerating within all of the different domains considered in this survey, it is imperative that a new updated survey is carried out in 5 to 10 years. By then it is expected that the tools will reach the quality of commercial-grade IDEs and integration with a wide variety of text editors [245], and the techniques will become a known factor when developing and designing a new system, application, or a hardware component.

4 Conclusions

More than 30 years ago, Burrows et al. published their pioneering work on the BAN logic for security protocol analysis [52]. Their work was not fully formal and was shown to permit approval of dangerous protocols. Nevertheless, they showed that their logic was good at revealing various subtle security flaws and drawbacks, specifically in authentication protocols. They set out to answer five questions:

(1)

Does this protocol work?

(2)

Can it be made to work?

(3)

Exactly what does this protocol achieve?

(4)

Does this protocol need more assumptions than another protocol?

(5)

Does this protocol do anything unnecessary?

Their important paper inspired a generation of security researchers to use FM to devise and analyse security protocols and to answer similar questions. Furthermore, in describing the benefits of formal verification for software engineering, Dijkstra famously quoted, “Testing shows the presence, not the absence of bugs.” More than 50 years later, in the setting of computer security, we might now have sufficient evidence to claim “Formal methods show the presence, not the absence of security flaws” [54]. That is, although FM provides rigorous tools and techniques for proving the absence of security flaws, this rigour comes with a proviso: Proofs are only possible if the security flaw is documented and specified within a formal framework. Current limitations mean that FM cannot uncover any new flaws, since we may not actually be looking for them.

Take for instance the recent Spectre and Meltdown attacks [145]. Like many vulnerabilities, both attacks have been shown to exist for a range of processors that make use of speculative execution, and mitigation against these requires software-side interventions. However, despite the widespread nature of these vulnerabilities, the attacks themselves were not discovered through formal verification, but rather through a series of experiments over the training and timing of micro-architectural components. Fortunately, once a security flaw has been uncovered, even complex attacks such as Spectre and Meltdown can be formally characterised and isolated [59]. Once this has been done, the next phase is a formal framework for reasoning about such issues, followed by more streamlined tools to scale verification to larger-scale systems. Thus, as important as it is to continue research into the practical use of FM in security, it is equally important to expand our reasoning capabilities for FM in security through the study of theoretical aspects of the discipline. Without this, there is a possibility of a new type of security flaw that falls outside the realms of current-day logics. Variants of Spectre and Meltdown, for instance, can be captured by subset-closed hyper-properties [59]. Hence, without existing works on hyper-properties [68] and subsequent works on their verification [59, 106, 107], the specification and therefore use of FM to protect against Spectre and Meltdown would have been much more difficult.

As for the utilisation of different FM techniques, we have uncovered that model checking is often a choice of practitioners when analysing wider systems. From the surveyed works, a trend could be seen where the model checking is often considered a good approach due to significant amount of tool support, plethora of modelling languages to choose from, the ability to carry out an exhaustive analysis, and even existence of model checkers specifically aimed at security analysis [200]. We could determine that even against problems where model checking would suffer from the size of the problem, the authors scope the problem such that it becomes analysed partially, specifically aiming at a critical parts of the system or an application. In some cases, either due to the large size of the problem or the problem’s suitability, the FM practitioners choose to utilise theorem proving. While the tool support is not on par with model checking and the learning curve is steeper, theorem proving provides exhaustive analysis of systems that would overwhelm model checkers. We are, however, observing that theorem proving needs to become more accessible to system and software engineers to increase its utilisation. Finally, a special category of lightweight FM provides an interesting entry point to the world of FM. On one hand, the inability to carry out exhaustive analysis could be seen as a downside, but on the other hand, it could be argued that lightweight FM are reaching ubiquity within the engineering world. For example, static code analysers are not only becoming used within software-build toolchains but often integrated directly to software-integrated development environments [221], being at the fingertips of significant portion of software developers around the world. As pondered in the Section 3, this could be one of the catalysts bringing the FM to the foreground in system design and development.

In this article, we have shown how FM have had an impact on society so far and how this impact will increase in the future. In the past, security has been an optional extra that industry does not want to invest in during development. But times are changing. For example, security has become a core selling point for Amazon Web Services (see Section 2.4). FM have been used successfully for security analysis in the financial, industrial, consumer, and enterprise sectors (see Section 2).

Specification Languages and Associated Tools

Our survey covers more than a decade of the use of FM in security. It reveals the rich variety of formal specification languages and their tools, theorem provers, model checkers, and verification frameworks. We have recorded more than 40 different specification languages and more than 40 different verification tools. These include the following:

Specification languages . AADL (Architecture Analysis & Design Language) [18, 70], ASF (Anonymous Secure Framework) [153], ASLan++ (AVANTSSAR Specification Language) [20], BAN logic [52, 228], Boogie [31], Boxed Ambients [135], CASM (ASM-based SL for compilers) [252], CCS (Calculus of Communicating Systems) [152], COVERT (compositional analysis of Android apps) [26], CSP (Communicating Sequential Systems) [120], CSP \(\#\) (shared variables CSP) [237], CTL (Computation tree temporal logic) [229], Cloud Calculus [135], Cryptol [89], Dynamic State Machine [187], ERC20 token contracts [199], Event-B [83], HLPSL (High Level Protocol Specification Language) [44], Hoare logic [108], LS2 (Logic of Secure Systems) [27], LTL (linear-time temporal logic) [266], Markov Decision Process [182], Petri nets [15], \(\pi\) -calculus [40], PlusCal [9], Promela [172], RTL (real-time logic) [110], SPDL (Security Protocol Description Language) [168], SysML-Sec [18], TLA+ (Temporal Logic of Actions) [72], Trusted CSP \(\#\) [27], überSpark [252], VDM [98], Verilog [164], VHDL [111], VML [226], vTRUST [117], XMHF (eXtensible and Modular Hypervisor Framework) [251], Z [265].

Model checkers . AVISPA (Automated Validation of Internet Security Protocols and Applications) [21], Alloy [83], CBMC (Bounded Model Checker for C and C++) [58], CWB-NC (Concurrency Workbench of New Century) [152], Cadence IFV (RTL block-level verifier) [111], FDR [103], GROOVE [102], jKind [99], NuSMV [65], OFMC (on-the-fly model checker) [34], PAT (Process Analysis Toolkit for CSP \(\#\) ) [237], PRISM (probabilistic model checker) [182], SATMC (SAT-based model checker for security protocols) [44], SPIN [247], TRUSTFOUND [27], UPPAAL [177], UVHM (formal analysis scheme for hypervisors) [251].

Theorem provers . Coq [72], Isabelle/HOL [144], K-framework [199], TAMARIN [154], Why [95].

Verification tools and frameworks. AndroBugs (Framework For Android Vulnerability Scanning) [241], Cl-Atse (protocol analyser) [141], FUDGE (Fuzz driver generator) [24], Frama-C [252], Krakatau [152], Maude (rewrite engine) [193], MobSF (mobile security framework) [127], phpSAFE [195], ProVerif [40], Quark [134], SAW (Software Analysis Workbench) [72], SMACK [72], SecGuru [136], SecVeriLog [92], Sugar (SAT-based) [169], TTool (translator from SysML-Sec to \(\pi\) -calculus) [18], Z3 [22].

This shows how research and application in FM for security has developed since Burrows et al.’s seminal paper [52]. Our survey concentrated in particular on the practical application of these techniques, especially on an industrial scale. Today, it seems inconceivable that a company would produce a commercial secure system without subjecting it to formal analysis. We also suspect that hackers use formal techniques to crack supposedly secure systems.

As a final statement, we need to acknowledge that a survey provides a snapshot in time within a developing field. It is therefore necessary to return to a survey work every decade, something we are planning on doing. Despite this shortcoming, it is the opinion of the authors that a survey work is an important part of the research, as it provides a starting point and a direction indicator for new and experienced practitioners looking for works under the large field of formal methods in security.

Acknowledgments

We would like to thank Nick Battle, Jaco van de Pol, and Bas Spitters for reviews of earlier versions of this article. We would also very much like to thank the anonymous reviewers of an earlier version of this article for their valuable input, which definitely improved it.

Footnotes

The levels of the Common Criteria are:

EAL1:	Functional testing.	EAL5:	Semi-formal design and testing.
EAL2:	Structural testing.	EAL6:	Semi-formally verified design and testing.
EAL3:	Methodical testing and checking.	EAL7:	Formally verified design and testing.
EAL4:	Methodical design, testing, and reviewing.

See spivey.oriel.ox.ac.uk/corner/Fuzz_typechecker_for_Z.

For comprehensive information on Tokeneer, see the AdaCore webpages www.adacore.com/tokeneer, where the entire project archive can be downloaded. AdaCore distributes the material generated by Altran under contract to the NSA under the terms of the Technology Transfer Agreement agreed between Praxis and the NSA. This material consists of all the core and support software for the Tokeneer ID Station, project documents, test cases derived from the system test specification, test tokens, and biometric data.

⁴

Diomidis Spinelli: www.spinellis.gr/blog/20081018/.

⁵

On the ARM platform, there is a further proof that the binary code that executes on the hardware is a correct translation of the C code for sel4. This means that the compiler does not have to be trusted, and extends the functional correctness property to the binary. See docs.sel4.systems/FrequentlyAskedQuestions.html.

⁶

github.com/Storyyeller/Krakatau.

⁷

Termed multi-process architecture in Google Chrome with sandboxing of untrusted code, which accesses resources through a trusted broker.

Appendix

This appendix provides an overview of the several surveyed research works in the form of a cross-tabulation according to Reference [166]. We group the 115 covered references by Domain, Level Of Abstraction (LOA) and Approach (App.).

Table 1.

References

[1]

Jasper RTL Apps. 2020. Cadence IFV Model Checker. Retrieved from www.cadence.com/en_US/home/tools/system-design-and-verification/formal-and-static-verification/jasper-gold-verification-platform.html.

Abstract

1 Introduction

1.1 Methodology

1.2 History

1.3 Definitions/background

2 Survey

2.1 Categorisation and Overview

2.2 Financial

2.3 Industrial

2.4 Consumer

2.5 Enterprise

3 Future Outlook

4 Conclusions

Specification Languages and Associated Tools

Acknowledgments

Footnotes

Appendix

References

Cited By

Index Terms

Recommendations

Government regulations in cyber security: Framework, standards and recommendations

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

Security Modeling and Analysis

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations