Chaos Engineering of Ethereum Blockchain Clients

There are several surveys that focus on blockchain [11, 16, 22, 30, 51]. For example, Huang et al. [30] conducted a survey of the state-of-the-art on blockchains including the theories, models, and tools. Based on these surveys, we notice that there is limited research on using fault injection techniques to evaluate the reliability of blockchain systems. Most of the recent works in this direction are about fuzzing smart contracts. To our knowledge, there is no work on perturbing the runtime environment of blockchain node, as is done by ChaosETH.

Regarding performance, the existing research works focus on different levels, including EVM opcode level [4, 6], smart contract level [5], consensus algorithms level [3, 28], blockchain system level [8, 9, 19, 20, 21], and blockchain-based application level [49]. Compared with these works, ChaosETH does not focus on performance evaluation of blockchain systems. Instead, performance-related metrics such as memory usage are used by ChaosETH to evaluate the side effects caused by system call invocation errors. The most related work is done by Dinh et al. [21], who invented BlockBench, a framework that evaluates a private blockchain’s performance using its throughput, latency, scalability, and fault-tolerance capability as indicators. BlockBench investigates how Byzantine failures affect a blockchain system’s throughput and latency by crashing some nodes, injecting network delays, and corrupting messages among different nodes. Compared with BlockBench, ChaosETH assesses a blockchain client using a public blockchain network, which means developers do not need to have the full control of the whole decentralized system. The failure models are also different between these two works.

Regarding reliability analysis and improvement, Seol et al. [42] implemented a blockchain analytics engine for assessing the dependability of off-chain file systems. Sousa et al. [45] designed a Byzantine fault-tolerant ordering service for Hyperledger Fabric. Zhang et al. [59] designed LedgerGuard, a mechanism that keeps the integrity of a ledger via corrupted blocks detection and recovery. Liu et al. [37] proposed an evaluation methodology that applies a continuous-time Markov chain model for blockchain-based IoT applications. It has been proposed to use modeling techniques to study blockchain reliability. For instance, Melo et al. [38] proposed a modeling methodology that evaluates reliability and availability of a blockchain-as-a-service environment. Kancharla et al. [33, 34] applied simulation methods to demonstrate the dependability of the proposed hybrid blockchain and slim blockchain. González et al. [26] categorized different fault injection techniques and discussed the possibilities to apply them in blockchain-based applications resilience assessment. Instead of using simulations or focusing on blockchain-based applications, ChaosETH uses real Internet traffic to assess the resilience of Ethereum clients in production.

Regarding security, research has been done on analyzing vulnerabilities that are located in smart contracts [7, 10, 18, 24, 27, 36, 50]. For example, Fu et al. [24] designed EVMFuzzer, a framework that generates contracts via a set of predefined mutators to find security bugs in different EVM implementations. Zhang et al. [58] presented TxSpector, a logic-driven approach for detecting attacks in Ethereum transactions at the bytecode level. Aumasson et al. [12] reviewed four Ethereum clients from a security perspective. Their work consists of a benchmarking methodology, as all of the four clients are evaluated using the same set of security problems. However, it is different from this article, because ChaosETH focuses on resilience benchmarking instead of security.

Moreover, none of these solutions support the specification and execution of fault injection experiments directly in a production-like environment. ChaosETH is the first methodology that provides developers with a systematic way to learn how their Ethereum client implementations react to different system call invocation errors in production.

Basiri et al. [13] presented the principles of chaos engineering in 2016. The earliest known chaos engineering tool is called the “Chaos Monkey” [13], which has never been deployed to Ethereum, as opposed to the “Bored Ape.” [2] Zhang et al. [57] designed ChaosMachine, a tool that conducts chaos engineering experiments at the try-catch level for Java applications. Jernberg et al. [32] designed a chaos engineering framework based on the literature and a tool survey and validated the framework in a real commercial web system. Simonsson et al. [43] proposed ChaosOrca, a chaos engineering system that injects system call errors for dockerized applications. Hernández-Serrato et al. [29] discussed the possibilities to apply machine learning techniques for improving chaos engineering experiments. Ikeuchi et al. [31] proposed a framework for learning a recovery policy using deep reinforcement learning and chaos engineering techniques. Chaos engineering is also applicable in the field of security. Torkura et al. [48] proposed CloudStrike, a tool that focuses on injecting failures that impact security, i.e., integrity, confidentiality, and availability. Regarding human factors in chaos enginering, Canonico et al. [15] discussed what aspects of AI would be used to make a system more resilient to perturbations and the results of these findings against existing chaos engineering approaches.

The only related work that combines chaos engineering and blockchains is by Sondhi et al. [44]. They apply chaos engineering to evaluate the performance of different consensus algorithms in permissioned blockchains. Their perturbation models were designed for representing network failures and message corruptions. In comparison, ChaosETH considers an entirely different failure model: errors at the level of system call invocations. This failure model captures the problems that may happen on the node’s operating system and not on the network. To our knowledge, our methodology of end-to-end chaos engineering for blockchain systems is novel.

As introduced in Section 2.2, defining the steady state of the Ethereum client under study is essential for a chaos engineering experiment. Per the state-of-the-art of observability, all Ethereum clients provide some monitoring capabilities. In ChaosETH, the steady state analyzer collects behavior-related metrics directly from the monitoring component provided by an Ethereum client.¹ For instance, the latest version of the GoEthereum client, v1.10.25, exposes more than 400 different metrics that describe the runtime status of the client. By reusing these pre-existing mechanisms, there is a guarantee that no extra monitoring overhead is introduced.

Recall the first challenge mentioned in Section 3.2: Not all the provided metrics are appropriate for describing the steady state. For instance, the highest block number is not an ideal metric for steady state inference, because this metric is determined by the whole blockchain network instead of the client itself. To address this challenge, the steady state analyzer first needs developers to define a set of metrics from the client’s monitoring module. Then, the analyzer conducts statistical analysis on the selected metrics and confirms if a metric is statistically stable for describing the client’s steady state.

For the selected metrics, the monitoring is done per “monitoring interval,” which is a period of time during which data is collected and persisted. The monitoring interval is set by developers and is an engineering tradeoff for developers between overhead and precision [56]. A shorter monitoring interval may produce a better picture of the steady state, but also means a larger amount of monitoring data and a higher monitoring overhead. A typical monitoring interval is 15 seconds.

For the steady state inference task, we first configure a “monitoring epoch” in the analyzer. A monitoring epoch is defined as a sequence of contiguous monitoring intervals. For example, if the monitoring epoch is 3,600 seconds and the monitoring interval is 15 seconds, then the analyzer records \(3,600 / 15 = 240\) data points for each metric. For a given configuration, the analyzer samples all metrics over two monitoring epochs. Then, for each metric, the samples from the two monitoring epochs are compared to each other using the Mann-Whitney U test. This statistical test is used to determine if the probability distribution of the two epochs is different. The stable metrics are those whose distributions over the two epochs are not statistically different and are selected to describe the client’s steady state.

After inferring the client’s steady state by means of metric distributions, ChaosETH can conduct chaos engineering experiments, where the core idea is to compare the metric distributions under error injection against the reference distributions.

To conduct chaos engineering experiments, we need to inject errors. In this article, we focus on system call errors and design a system call error injector accordingly. When the injector is activated, it listens to system call invocation events, and it replaces the original return code with an error code. In ChaosETH, the error injector uses an error injection model defined as a triple (s, e, r): It means that system call s is injected with error code e under the error rate r \(\in [0,1]\) . The error rate is the probability of replacing the return code by error code e when system call s is invoked at runtime. For example, an error injection model (read, EAGAIN, 0.5) means that every time a read system call is invoked, there is a \(50\%\) probability that a successful return code is replaced with error code EAGAIN, which represents that the resource being read is temporarily unavailable [46].

Depending on the type of system call to perturb and the error rates, the error scenario is more or less severe. The higher the error rate in an error injection model, the more frequently such errors are injected into the target client, which means that the client is subject to a higher resilience pressure.

Considering the second challenge mentioned in Section 3.2, ChaosETH aims at focusing on realistic errors. For this, we follow the state of the work by Zhang et al. [56]. According to their methodology, realistic fault models should fulfill two characteristics: They represent real production failure scenarios and happen frequently enough so developers are able to analyze the behavior during a chaos engineering experiment. Consequently, in ChaosETH our fault models are realistic because: (1) They use the same error code as system call invocation errors that are observed in the field, and (2) the error rate is amplified so more invocation errors can be observed in a short period of time and trigger resilience issues.

The experiment orchestrator communicates with the steady state analyzer and the system call error injector to conduct chaos engineering experiments. The orchestrator attaches these two components to the operating system kernel before the Ethereum client starts. Then it activates the error injector according to the experiment configuration. An experiment configuration defines the duration of an experiment and the corresponding error injection model to be used.

The orchestrator divides each chaos engineering experiment into five phases: (1) the warm-up phase, (2) the pre-checking phase, (3) the error injection phase, (4) the recovery phase, and (5) the validation phase. During the warm-up phase, the target client runs until it reaches its steady state. During the pre-checking phase, the steady state analyzer is activated to monitor the client and to check if the client has already reached its steady state. During the error injection phase, both the steady state analyzer and the error injector are turned on so system call invocation errors are injected according to the given error injection model. After the error injection phase is done, the orchestrator gives the target client time to recover during a recovery phase. After this recovery phase, the validation phase takes place during which the orchestrator monitors the client’s state again. The observed behavior during this validation phase is compared with the previously inferred steady state using the Mann–Whitney U test. If the target client fails to recover back to its steady state after the validation phase, then it means that the injected errors have caused an adverse side effect on the client. This can signify to the developers that the client’s error handling mechanisms may have failed to recover from the injected error.

After a chaos engineering experiment, the orchestrator analyzes the client’s behavior during the experiment and generates a resilience report for developers. The report presents to what extent the implementation of the Ethereum client under evaluation is resilient to the injected system call invocation errors. As introduced in Section 2.2, the client’s resilience is assessed by validating chaos engineering hypotheses. ChaosETH uses three hypotheses about how a client reacts when errors are injected using one specific error model.

Non-crash hypothesis ( \(H_N\) ). The non-crash resilience hypothesis holds if the injected system call invocation errors do not crash the Ethereum client and the process remains alive.

Observability hypothesis ( \(H_O\) ). The observability hypothesis for a metric m holds if m is influenced by error injections according to an error model.

Recovery hypothesis ( \(H_R\) ). The recovery hypothesis is valid if the client is able to recover to its steady state after stopping the injection of system call invocation errors.

For example, let us assume that the system call error injector follows the error model (read, EAGAIN, 0.5) to conduct the experiment. The target client does not crash during fault injection. However, the success rate of reading data from disk drops significantly during fault injection, because the client fails at invoking the read system call. Furthermore, the success rate of reading data gets back to its steady state again after the experiment stops. In this case, ChaosETH reports that all of the three hypotheses \(H_N\) , \(H_O\) , and \(H_R\) are validated, meaning that the target client is non-crashing, observable, and resilient with respect to error model (read, EAGAIN, 0.5). By validating or falsifying the hypotheses above, developers learn more about the Ethereum client’s resilience with respect to different types of system call invocation errors. Such information is helpful for developers to prioritize their work on improving error handling.

The system call monitor and system call error injector are based on the Phoebe framework [56], which implements them with the help of the eBPF (extended Berkeley Packet Filter) module. More specifically, the monitor and error injector register their BPF programs to the sys_enter and sys_exit events. When there is an error that needs to be injected, the error injector calls the BPF helper function bpf_override_return to replace the original return code with the error code. The experiment orchestrator is implemented in Python. The source code of ChaosETH is publicly available at https://github.com/KTH/royal-chaos/tree/master/chaoseth

To answer RQ1, we conduct a steady state analysis experiment on each client. As introduced in Section 2.2, characterizing an Ethereum client’s steady state is a prerequisite for a chaos engineering experiment. Conducting steady state analysis gives developers a clear understanding of how the Ethereum client behaves in a normal production condition.

The first row of Figure 2 illustrates our methodology to model the steady state. Each Ethereum client (GoEthereum or Nethermind) is started up using the options recommended in their respective documentation. The clients are started in active synchronization mode, which consists in downloading and verifying the blocks from other peers on the Internet. After starting up the client, we wait for the client to finish synchronizing the existing blocks.

After a client reaches a synced state, the main job of the client is to verify newly generated blocks and to share all the blocks with other peers. From this point, the steady state analyzer is attached to the client with a monitoring interval of 15 seconds and a monitoring epoch of 5 hours, per our conceptual framework detailed in Section 3.4.1. These values ensure that a sufficient number of data points ( \(5\times 60\times 60\div 15=1,200\) points) are available for capturing the distribution of each metric.

A steady state analysis experiment takes all of a client’s exposed metrics as input and outputs each metric’s probability distribution. The metrics that meet the following two criteria are selected for chaos engineering experiments: (1) being an active non-zero metric (some metrics are always zero if a feature is not turned on) and (2) being statistically stable to describe the client’s steady state.

To answer RQ2, we conduct chaos engineering experiments on each client. During a chaos engineering experiment, all of the components in ChaosETH are activated. The system call error injector intercepts a specific type of system call invocation. It replaces the successful return code with an error code, while the Ethereum client is connected to the rest of the blockchain distributed system. To perform the chaos engineering experiment, recall that we use the results of the steady state analysis (see Section 3.4.3), and we use those of RQ1. Also, the duration of the warm-up phase is 2 hours. The pre-check, error injection, and validation phases of a chaos engineering experiment in Figure 2 are set to 5 minutes for each phase. The recovery phase between the error injection phase and the validation phase is set to 10 minutes to give a client more time for self-recovery.

RQ3 is about resilience benchmarking experiments, which aim at providing insights for end-users who wish to choose a suitable Ethereum client, based on resilience requirements. Our benchmarking experiments indicate which client has a better resilience against some system call invocation errors. For a resilience benchmarking experiment, we define a single, common error model for all of the target clients to have a sound comparison. A common error model (s, e, \(r_m\) ) is one error type (s, e, \(r_c\) ) that has been observed on each client in isolation. The common error rate \(r_m\) is defined as the maximum value of \(r_c\) . The last row in Figure 2 illustrates our procedure for resilience benchmarking. Once both of the clients reach their steady state, ChaosETH conducts chaos engineering experiments using the common error model on each client. A cross-comparison is made among each client’s resilience report that is generated in the chaos engineering experiment. For example, ChaosETH injects an ENOMEM (insufficient memory) error into a read system call invocation. If client A directly crashes while client B continues to run and conducts retries after this error is injected, then client B is more resilient with respect to the injected type of errors.

We perform a steady state analysis experiment on each selected Ethereum client, per Section 4.2.1. Every client is observed for two monitoring epochs of 5 hours, amounting to 10 hours in total. Within each epoch, the metrics of interest are recorded and aggregated for every 15-second monitoring interval.

Figure 3 depicts the distributions of metrics in GoEthereum and Nethermind. In the case of GoEthereum, the client exposes 400 metrics in total. ChaosETH analyzes all of them to identify proper metrics for chaos engineering experiments. As not all of the client’s features are activated using the recommended configuration, 164 out of these 400 metrics are inactive, meaning that the value of these metrics never change or cannot be queried. The comparison of the distributions of active metric values in two different monitoring epochs shows evidence that 44 metrics are statistically stable. As mentioned in Section 3.4.1, the steady state analyzer utilizes the Mann Whitney U test for distribution comparison. In all of the experiments, we use a p-value of 0.03. Under a confidence level of 0.01, the null hypothesis that the two samples are not statistically distinguishable is not rejected. In the case of Nethermind, 231 metrics are analyzed by the steady state analyzer, 115 metrics are inactive during the experiment. Regarding the 116 active metrics, 55 are statistically stable and can be used for further experiments.

In Table 1, we display the evolution of metric samples during the steady state experiment. The first half of each evolution chart (in blue) is based on the data gathered during the first monitoring epoch. The second half of the chart (in red) is drawn based on the data of the second monitoring epoch. The last two columns indicate the p-values, obtained after applying the Mann–Whitney U test, to test for the similarity between the two distributions, and the result of the test. For example, the first row in Table 1 shows that the number of account flush operations made by the GoEthereum client regularly has spikes during these two monitoring epochs.

An example where the null hypothesis is rejected at confidence level 0.01 is metric json.rpc.requests(count/s) in the Nethermind client. From Table 1 the line chart in the second-to-last row also visually confirms that the metric does not evolve in the same way during the two monitoring epochs. Considering the confidence level of 0.01, this metric is not stable enough to describe a client’s steady state and thus is excluded from further experiments.

This experiment shows that not all the monitoring metrics provided by an Ethereum client are suitable to describe the client’s steady state in a statically valid manner. Since the experiments are done in production, there are several factors that could affect a metric’s stability. First, the node itself is not always stable. For instance, there exist other applications that take more resources from the node. Second, the network may not be stable. It is possible that the node encounters network scans or attacks every now and then [16]. Last, the behaviors of peers are different. For example, when the node randomly connects with some new peers who have different characters, a metric might be influenced.

From the experiment for RQ1, we know that GoEthere-um runs with 10 different types of system calls, accumulating more than 288 million invocations in a 10–hour production run (two monitoring epochs). Interestingly, none of the types of system call invocations has a 100% success rate. We perform chaos engineering by increasing the error rates of those system calls in production. The error rate amplification approach described in Section 3.4.2 produces 15 and 12 realistic error models, respectively, for the GoEthereum client and the Nethermind client. For each error model, we make a chaos engineering experiment with a one-to-one mapping.

Table 2 describes the error models together with the chaos engineering experiments of the selected clients. Every row presents one error injection model, including the target system call invocation (column Syscall), the error code to be injected, and the error rate. The last five columns give the corresponding experiment result, including the total number of injected errors, the number of evaluated metrics, and the results of whether the three hypotheses ( \(H_N\) , \(H_O\) , \(H_R\) ) are verified or falsified with respect to a metric. The metrics that fail the pre-check phase are excluded from the other phases, since ChaosETH considers them not stable enough for behavior comparison. When the client does not invoke a type of system call during the experiment, ChaosETH does not inject any error related to that type of system call, and the corresponding row is omitted in the table.

For the GoEthereum client, ChaosETH conducts 12 chaos engineering experiments. The results show that 5 out of 12 error models crash the GoEthereum client (the rows whose \(H_N\) column is marked with “X”). For the other 7 error models, 6 of them have a visible effect on the monitoring metrics (the rows whose \(H_O\) column contains a non-zero value). For example, when ChaosETH uses the error model (accept, EAGAIN, 0.6) for experiments, 24 metrics are stable during the pre-check phase. During the error injection phase, 18 metrics are observed to deviate from their normal behavior. After stopping injecting the errors, 16 out of these 18 metrics recover to the normal state after the recovery phase. This confirms that the GoEthereum client is resilient to EAGAIN errors in accept4 with respect to these 16 metrics.

Regarding the Nethermind client, there are 10 chaos engineering experiments in total (second half of Table 2). The results show that two error models, (futex, EAGAIN, 0.05) and (futex, ETIMEDOUT, 0.05), lead the Nethermind client to a crash. Seven error models cause a visible effect on at least one metric during the error injection phase. The error model (recvfrom, EAGAIN, 0.549) does not cause any impact on all of the 48 metrics that pass the pre-check. In this case, ChaosETH does not check the \(H_O\) hypothesis, because no metric deviates from its steady state even during the error injection phase.

This experiment has five main outcomes, with different meanings for the Ethereum developers.

Crash ( \(H_N\) =X). The client directly crashes because of the injected errors. This is considered as a severe case: This means that an Ethereum node disappears from the distributed consensus and validation process. As the client crashes, the hypotheses \(H_O\) and \(H_R\) cannot be tested and are marked as “-” in Table 2. For example, ChaosETH detects that the GoEthereum client directly crashes when an EAGAIN error code is injected to the system call write. Since error code EAGAIN in Linux means that the target resource is temporarily unavailable, crashing is an over-reaction; the client should consider implementing a classical retry mechanism instead of crashing directly.

Invisible effect ( \(H_N\) = \(\checkmark\) and \(H_O\) =0). In some cases, there is no visible effect detected during the error injection phase. For example, the Nethermind chaos experiment using error model (recvfrom, EAGAIN, 0.549) reveals such a situation. In this experiment, ChaosETH injects 51,715 system call invocation errors to system call recvfrom. During this error injection phase, none of the 48 metrics have an abnormal behavior. This indicates that the Nethermind client seems to be functioning normally when a system call invocation to recvfrom returns an EAGAIN error code, which can potentially signify resilience. However, we cannot exclude that the client state is corrupted in an invisible manner, because we do not have a provably perfect steady state oracle. Since ChaosETH does not capture anything abnormal during the error injection phase, the verification of hypothesis \(H_R\) is skipped. Overall, the presence of such invisible effect cases is good with respect to consistency: If we would not perform steady state pre-checking and observability hypothesis checking, then developers may falsely believe that the client state is valid according to the monitored metrics.

Long-term effect ( \(H_N\) = \(\checkmark\) , \(H_O\) = \(\checkmark\) , and \(H_R\) =X). For some of the error models, the client under experiment does not crash. However, during the error injection phase, some metrics deviate from their steady state and do not recover after the given recovery phase. For instance, the experiment result of error model (accept4, EAGAIN, 1) in the Nethermind client belongs to this category. During the error injection phase, it shows that metrics eth66get_block_headers_received/s, local_receive_message_timeout_disconnects/s, process_private_memory/s, and process_virtual_memory/s deviate from their normal behavior. However, after the recovery phase, only metrics process_private_memory/s and process_virtual_memory/s recover to the steady state. The other two metrics stay abnormal during the validation phase. This means that either it takes a longer time for the client to recover from the injected errors or that the injected errors lead the client to a stalled or corrupted state. Overall, such cases show that ChaosETH gives Ethereum developers insights about the timespan of recovery.

Resilient case ( \(H_N\) = \(\checkmark\) , \(H_O\) = \(\checkmark\) , and \(H_R\) = \(\checkmark\) ). Certain error models do not crash the client and also cause visible evidence of resilience. After the error injection stops, the monitoring metrics recover to their steady state. This indicates that the target client is equipped with an effective, graceful error-handling mechanism that brings the client back to normal after errors. For example, during the chaos engineering experiment using error model (connect, EINPROGRESS, 0.8) in the GoEthereum client, the injected errors do not crash the client, thus the \(H_N\) hypothesis holds. During the error injection phase, the metric geth.txpool.slots.gauge/s no longer matches the steady state. When the error injection stops, the client’s behavior related to the transaction pool slots is restored during the recovery phase. During the validation phase, ChaosETH checks the metric again and confirms that geth.txpool.slots.gauge/s has recovered to its steady state. By looking at the client logs, we indeed confirm that the client has resumed downloading, sharing, and verifying Ethereum blocks.

We cannot strictly compare the considered clients based on the results of RQ2, because the error models are different. To overcome this, we have introduced in Section 4.2.3 the idea of testing the clients under a meaningful common error model. ChaosETH identifies four common error models for the selected clients. The results of this resilience benchmarking experiment are summarized in Table 3. Each row in the table presents the verification of the three hypotheses for both clients, according to a set of client metrics. Only the metrics that pass the pre-check phase are selected for hypothesis verification. This table is interesting in the following three aspects:

First, regarding the \(H_N\) hypothesis (absence of crash), the results show that both the GoEthereum client and the Nethermind client crash under the same specific error models. These two clients are crashed by futex system call invocation errors with codes EAGAIN and ETIMEDOUT. Overall, there is no client that is absolutely more robust than the other with respect to crashing.

Second, focusing on the \(H_O\) hypothesis (observability), when the error model (accept4, EAGAIN, 1) is used for experiments, both the GoEthereum client and the Nethermind client are observed to have abnormal behavior with respect to metrics. For the GoEthereum client, 9 metrics become abnormal during the error injection phase. Regarding the Nethermind client, 6 metrics deviate from the steady state. This is evidence that the metrics capture the client’s internal state and that not all clients have the same observability.

Third, considering the \(H_R\) hypothesis, ChaosETH successfully identifies resilient cases for the two Ethereum clients. ChaosETH shows that the GoEthereum client is resilient to error model (accpet4, EAGAIN, 1) with respect to metrics geth.p2p.peers.gauge/s and geth.txpool.reheap.timer/s. For the same error model, the Nethermind client is resilient with respect to metrics nethermind_mod_exp_precompile/s, nethermind_state_db_reads/s, and nethermind_useless_peer_disconnects/s. As opposed to toy examples with perfect oracles, assessing behavior of real-world software through monitoring yields multiple shades of resilience.