A Model and Survey of Distributed Data-Intensive Systems

Atomicity ensures that a group of tasks either entirely succeeds or entirely fails. We use the established jargon of database transactions and say that a task (or group of tasks) either commits or aborts. If the tasks commit, all the effects of their execution, and particularly all their changes to the shared state, become visible to other tasks. If the tasks abort, none of the effects of their execution becomes visible to other tasks. We classify group atomicity along two dimensions. First, we consider the possible causes for aborts and distinguish between system-driven or job-driven. System-driven aborts (e.g., a worker running out of memory) derive from non-deterministic hardware or software failures, whereas job-driven aborts (e.g., database integrity constraints) are part of a job definition and are triggered if job completion may lead to a logic error. Second, we consider how systems implement group atomicity. Atomicity is essentially a consensus problem [65], where tasks need to agree on a common outcome: commit or abort. The established protocol to implement atomicity is two-phase commit. In this protocol, one of the participants (tasks) takes the role of a coordinator, collects all votes (commit or abort) from participants (phase 1), and distributes the common decision to all of them (phase 2). Notice that this protocol is not robust against failures of the coordinator; however, data-intensive systems typically adopt orthogonal mechanisms to deal with failures, as discussed in Section 2.7. Most importantly, two-phase commit is a blocking protocol as participants cannot make progress before receiving the global outcome from the coordinator. For these reasons, some systems adopt simplified, coordination-free protocols, which reduce or avoid coordination under certain assumptions. Being specific to individual systems, we discuss such protocols in Section 4.

Group isolation constrains how tasks belonging to different groups can interleave with each other and is classically organized into levels [4]. The stronger, serializable isolation, requires the effects of execution to be the same as if all groups were executed in some serial order, with no interleaving of tasks, whereas weaker levels enable some disciplined form of concurrency that may lead to anomalies in the results that clients observe. In line with the approach adopted for replication consistency and for atomicity, in this work we consider only two broad classes of isolation levels: those that require blocking coordination between tasks and those that are coordination free (referred to as being highly available in the literature [15]). This is also motivated by the systems under analysis, which either provide strong isolation levels (typically, serializable) or do not provide isolation at all.

Implementation-wise, strong isolation is traditionally achieved with two classes of coordination protocols: lock-based and timestamp-based. With lock-based protocols, tasks acquire non-exclusive or exclusive locks to access shared resources (shared state in our model) in read-only or read-write mode. Lock-based protocols may incur distributed deadlocks: to avoid them, protocols implement detection or prevention schemes that abort and restart groups in the case of deadlock. Timestamp-based protocols generate a serialization order for groups before execution, then the tasks need to enforce that order. Pessimistic timestamp protocols abort and re-execute groups when they try to access shared resources out of order. Multi-version concurrency control protocols reduce the probability of aborts by storing multiple versions of shared state elements and allowing tasks to read old versions when executed out of order. Optimistic concurrency control protocols allow out-of-order execution of tasks but check for conflicts before making the effects of a group of tasks visible to other tasks. Finally, a few systems adopt special protocols that reduce or avoid coordination under certain assumptions: as in the case of group atomicity, we discuss these protocols in Section 4.

Functional Model. All DMSs provide a global shared state that applications can simultaneously access and modify. In the more traditional systems, there is a sharp distinction between the application logic (the driver, executed client-side on registration) and the queries (the jobs, executed by the DMS). Recent systems increasingly allow to move the part of the application logic that orchestrates jobs execution within the DMS, in the form of stored procedures that run system-side on start. Stored procedures may bring two advantages:

(i) reducing the interactions with external clients, thus improving latency, and (ii) moving part of the overhead for compiling jobs from driver execution time to driver registration time.

Being conceived for interactive use, all DMSs offer synchronous APIs to invoke jobs from the driver program. Many also offer asynchronous APIs that allow the driver to invoke multiple jobs and receive notifications of their results when they terminate. A common approach to reduce the cost of communication when starting jobs from a client-side driver is batching multiple invocations together, which is offered in some NoSQL systems such as MongoDB [32] and Redis [64].

Several DMSs can interact with active sources and sinks. Active sources push new data into the system, leading to insertion or modification of state elements. Sinks register to state elements of interest (e.g., by specifying a key or a range of keys) and are notified upon modification of such elements.

DMSs greatly differ in terms of deployment strategies, which are vastly influenced by the coordination protocols that govern replication, group atomicity, and isolation. NoSQL systems do not offer group atomicity and isolation. Those designed for cluster deployment typically use blocking (synchronous or semi-synchronous) replication protocols. Those that support wide-area deployments either use coordination-free (asynchronous) replication protocols that reduce durability and consistency guarantees or employ a hybrid strategy, with synchronous replication within a data center and asynchronous replication across data centers. Many NewSQL systems claim to support wide-area deployments while offering strong consistency, group atomicity, and isolation. We review their implementation strategies to achieve this result in Section 4.3.

Jobs. All DMSs implement one-shot jobs with explicit state management primitives to read and modify a global shared state. Jobs definition APIs greatly differ across systems, ranging from pure key-value stores that offer CRUD (create, read, update, and delete) primitives for individual state elements to expressive domain-specific libraries (e.g., for graph computation) or languages (e.g., SQL for relational data). In almost all DMSs, the execution plan and the communication between tasks are implicit and do not include iterations or dynamic creation of new tasks. A notable exception are graph databases, which support iterative algorithms where developers explicitly define how tasks update portions of the state (the graph) and exchange data. The structure of the execution plan also varies across systems: common structures include the use of a single task that implements CRUD primitives, workflows orchestrated by a central coordinator task, or hierarchical structures. Jobs are compiled on driver execution, except for those systems where part of the driver program is registered server-side (as stored procedures). Job compilation always uses static information about resources, such as the allocation of shared state portions onto nodes. Some structured NewSQL DMSs such as Spanner [14] and CockroachDB [90] also exploit dynamic information about resources—for instance, to configure a given task (e.g., select a sort-merge or a hash-based strategy for a join task) depending on some resource utilization or statistics about state (e.g., cardinality of the tables to join).

All DMSs perform deployment at the job level, when the job is compiled, with the only exception of AsterixDB [9], which compiles jobs into a dataflow plan and deploys individual tasks when they are activated [19]. Deployment is always guided by the location of the state elements to be accessed, which we consider as a static information. Indeed, under normal execution, shared state portions do not move, and our model captures dynamic relocation of shared state portions (e.g., for load balancing) as a distinct aspect (dynamic reconfiguration). In addition, we keep saying that deployment is based on static information even for those systems that exploit dynamic information (e.g., the load of workers) but only to deploy the tasks that manage the communication between the system and external clients. Finally, DMSs are not typically designed to operate in scenarios where the compute infrastructure is shared with other software applications. This is probably due to the interactive and short-lived nature of jobs, which would make it difficult to predict and adapt the demand of resources in those scenarios. The only case in which we found explicit mention of a shared platform is in the description of the Google infrastructure, where DMS components (BigTable [29], Percolator [76], Spanner [34]) share the same physical machines and their scheduling, execution, and monitoring is governed by an external resource management service.

Data and State Management. The data model (i.e., the structure of data and state elements) is a key distinguishing characteristic of DMSs, which we use to organize our discussion in Sections 4.2 and 4.3. Some systems explicitly consider the temporal dimension of data elements. For instance, some wide columns stores associate timestamps to elements and let users store multiple versions of the same element, whereas time-series databases are designed to efficiently store and query sequences of measurements over time. DMSs differ in terms of storage medium and structure. We detail the choices of the different classes of systems in Section 4.2 and Section 4.3, but we can identify some common concerns and design strategies. First, the representation of state on storage is governed by the data model and the expected access pattern: relational tables are stored by row, whereas time series are stored by column to facilitate computations on individual measurements over time (e.g., aggregation, trend analysis). Second, there is a tension between read and write performance: read can be facilitated by indexed data structures such as B-trees, but they incur higher insertion and update costs. Hierarchical structures such as LSM trees improve write performance by buffering updates in higher-level logs (frequently in-memory) that are asynchronously merged into lower-level indexed structured, at the expense of read performance, due to the need to navigate multiple layers. Third, most DMSs exploit main memory to reduce data access latency. For instance, systems based on LSM trees store the write buffer in memory. Similarly, most systems that use disk-based storage frequently adopt some in-memory caching layer. Finally, some systems adopt a modular architecture that supports different storage layers. This is common in DMSs offered as a service in public cloud environments (e.g., Amazon Aurora [95]) or in private data centers (e.g., Google Spanner [34]), where individual system components (including the storage layer) are themselves services.

Tasks always communicate using direct, ephemeral connections, which implement a partitioned, non-replicated data bus. Shared state is always partitioned across workers to enable concurrent execution of tasks. Most DMSs also adopt state replication to improve read access performance, with different guarantees in terms of consistency between replicas: NoSQL databases provide weak (or configurable) consistency guarantees to improve availability and reduce response time. NewSQL databases provide strong consistency using leader-based or consensus protocols and restricting the types of transactions (jobs) that can read from non-leader replicas—typically snapshot transactions, which are a subset of read-only transactions that read a consistent version of the state, without guarantees of it being the most recent one. Group atomicity and isolation are typically absent or optional in NoSQL databases, or restricted to very specific cases, such as jobs that operate on single data elements. Instead, NewSQL databases provide strong guarantees for atomicity and isolation, at the cost of blocking coordination. The transactional semantics of NewSQL systems ensures exactly once delivery. Indeed, transactions (jobs) either complete successfully (and their effects become durable) or abort, in which case they are either automatically retried or the client is notified and can decide to retry them until success. Conversely, NoSQL systems frequently offer at most once semantics, as they do not guarantee that the results of job execution are safely stored on persistent storage or replicated. In some cases, users can balance durability and availability by selecting the number of replicas that are updated synchronously. Finally, systems that support timestamps use event time semantics, where timestamps are associated with state elements by clients, whereas none of the systems provides order guarantees. Even in the presence of timestamps, DMSs do not implement mechanisms to account for elements produced or received out of timestamp order.

Fault Tolerance. Frequently, DMSs offer multiple mechanisms for fault tolerance and durability that administrators can enable and combine depending on their needs. Fault detection can be centralized (leader-worker) or distributed, depending on the specific system. Fault recovery mostly targets the durability of shared state. Since jobs are lightweight, DMSs simply abort them in the case of failure and do not attempt to recover (partial) computations. Transactional systems guarantee group atomicity: in the case of failure, none of the effects of a job become visible. To enable recovery of failed jobs, they either notify the clients about an abort, allowing them to restart the failed job, or restart the job automatically. Almost all DMSs adopt logging mechanisms to ensure that the effects of jobs execution on shared state are durable. Logging enables changes to be recorded on some durable append-only storage before being applied to shared state: most systems adopt a WAL that records changes to individual data elements, whereas few others adopt a command log that stores the operations (commands) that perform the change. Individual systems make different assumptions on what they consider as durable storage: in some cases the logs are stored on a single disk, but more frequently they are replicated (or saved on third-party log services that are internally replicated). Logging is frequently used in combination with replication of shared state portions on multiple workers: in these cases, each worker stores its updates on a persistent log to recover from software crashes, whereas replication on other workers may avoid unavailability in the case of hardware crashes or network disconnections. In addition, most systems offer geo-replication for disaster recovery, where the entire shared state is replicated in a different data center and periodically synchronized with the working copy. Similarly, many systems provide periodic or manual checkpointing to store a copy of the entire database at a given point in time. Depending on the specific mechanisms adopted, DMSs provide either no guarantees for state, such as in the case of asynchronous replication, or same state guarantees, such as in the case of persistent log or consistent replication.

Dynamic Reconfiguration. Most DMSs support adding and removing workers at runtime, and migrating shared state portions across workers, which enable dynamic load balancing and scaling without restarting the system. All systems support dynamic reconfiguration as a manual administrative procedure, and some can also automatically migrate state portions for load balancing. A special case of reconfiguration for NewSQL systems that rely on structured state (in particular, relational systems) involves changing the state schema: many systems support arbitrary schema changes as long-running procedures that validate state against the new schema and migrate it while still serving clients using the previous schema. Instead, systems such as VoltDB [89] rely on a given state partitioning scheme and prevent changes that violate such scheme.

Using an established terminology, we classify as NoSQL all those DMSs that aim to offer high availability and low response time by relinquishing features and guarantees that require blocking coordination between workers. In particular, they typically do the following. First, they avoid expressive job definition APIs that may lead to complex execution plans (as in the case of SQL, hence the name) [85]. In fact, the majority of the systems we analyzed focus on jobs comprising a single task that operates on an individual element of the shared state. Second, they use asynchronous replication protocols: depending on the specific protocol, this may affect consistency, may affect durability (if replication is used for fault tolerance), and may generate conflicts (when clients are allowed to write to multiple replicas). Third, they abandon or restrict group guarantees (atomicity and isolation) when jobs with multiple tasks are supported. In our discussion, we classify systems by the data model they offer.

NewSQL systems aim to provide transactional semantics (group atomicity and isolation), durability (fault tolerance), and strong replication consistency while preserving horizontal scalability. Following the same approach we adopted in Section 4.2, we organize them according to their data model.

Functional Model. Most DPSs use a leader-workers architecture, where one of the processes that compose the system (denoted the leader) has the special role of coordinating other workers. Such systems always allow submitting the driver program to the leader for system-side execution. Some of them also allow client-side driver execution, such as Apache Spark [99] and Apache Flink [24]. Other systems, such as Kafka Streams [18] and timely dataflow [71], are implemented as libraries where client processes also act as workers. Developers start one or more client processes, and the library handles the distributed execution of jobs onto them. Stream processing systems support asynchronous invocation of (continuous) jobs, whereas batch processing systems may offer synchronous or asynchronous job invocation APIs, or both. All DPSs support sources and sinks, as they are typically used to read data from external systems (sources), perform some complex data analysis and transformation (jobs), and store the results into external systems (sinks). Sources are passive in the case of batch processing systems and active in the case of stream processing systems. Most batch processing systems are stateless: output data is the result of functional transformations of input data. Stream processing systems can persist a (task) state across multiple activations of a continuous job. We model iterative graph algorithms as continuous jobs where tasks (associated with vertices, edges, or sub-graphs) are activated at each iteration and store their partial results (values associated with vertices, edges, or sub-graphs) in task state. DPSs assume a cluster deployment, as job execution typically involves exchanging large volumes of data (input, intermediate results, and final results) across workers.

Jobs. All dataflow systems provide libraries to explicitly define the execution plan of jobs. Increasingly often, they also offer higher-level abstractions for specific domains, such as relational data processing [2, 12, 23], graph computations [46], or machine learning [69]. Some of these APIs are declarative in nature and make the definition of the execution plan implicit. Task communication is always implicitly defined and controlled by the system’s runtime. Concerning jobs, dataflow systems differ with respect to the following aspects. The first is generality. MapReduce [38] and some early systems derived from it only support two processing stages with fixed operators, whereas later systems like Spark support any number of processing stages and a vast library of operators. The second is support for iterations. Systems like HaLoop [21] extended MapReduce to efficiently support iterations by caching data accessed across iterations in workers. Spark inherits the same approach and, together with Flink, supports some form of iterative computations for streaming data. Timely dataflow [71] generalizes the approach to nested iterations. The third is dynamic creation of tasks. Among the systems we analyzed, only CIEL [72] enables dynamic creation of tasks depending on the results of processing.

Data parallelism is key to dataflow systems, and all operators in their jobs definition API are data parallel. Jobs are one-shot in the case of batch processing and continuous in the case of stream processing. In the latter case, jobs may implicitly define some task state, such as by expressing computations that operate on a recent portion (window) of data rather than on individual data elements. Jobs cannot control task placement explicitly, but many systems provide configuration parameters to guide placement decisions—for instance, to force or inhibit the colocation of certain tasks. An exception to the preceding rules is represented by graph processing systems, which are based on a programming model where developers define the behavior of individual vertices [68]: the model provides explicit primitives to access the state of a vertex and to send messages between vertices (explicit communication).

All DPSs compile jobs on driver execution. For other characteristics related to jobs compilation, deployment, and execution, we distinguish between systems that perform task-level deployment (discussed in Section 5.2) and systems that perform job-level deployment (discussed in Section 5.3).

Data and State Management. Deployment and execution strategies affect the implementation of the data bus. In the case of job-level deployment, the data bus is implemented using ephemeral, push-based communication channels between tasks (e.g., direct TCP connections). In the case of task-level deployment, the data bus is mediated and implemented by a persistent service (e.g., a distributed filesystem or a persistent message queuing system) where upstream tasks push the results of their computation and downstream tasks pull them when activated. A persistent data bus can be replicated for fault tolerance, as in the case of Kafka Streams, which builds on replicated Kafka topics. CIEL [72] and Dryad [51] support hybrid bus implementations, where some connections may be direct while others may be mediated. Data elements may range from arbitrary strings (unstructured data) to specific schemas (structured data). The latter offer opportunities for optimizations in the serialization process, such as allowing for better compression or for selective deserialization of only the fields that are accessed by a given task. In general, DPSs do not provide shared state. Stream processing and graph processing systems include a task state to persist information across multiple activations of a continuous job (i.e., windows). In the absence of shared state, DPSs do not provide group atomicity or isolation properties. Almost all systems provide exactly once delivery, under the assumption that sources can persist and replay data in the case of failure and sinks can distinguish duplicates. The concrete approaches to provide such guarantee depend on the type of deployment (task level or job level) and are discussed later in Sections 5.2 and 5.3. Order is relevant for stream processing systems: with the exception of Storm [94], all stream processing systems support timestamped data (event or ingestion time semantics). Most systems deliver events in order, under the assumptions that sources either produce data with a pre-defined maximum delay or inform the system about the progress of time using special metadata denoted as watermark. Kafka Streams takes a different approach: it does not wait for out-of-order elements and immediately produces results. In the case in which new elements arrive out of order, it retracts updates the previous results.

Fault Tolerance. All DPSs detect faults using a leader-worker architecture, and in absence of a shared state, they recover from failures through the mechanisms that guarantee exactly once delivery.

Dynamic Reconfiguration. DPSs use dynamic reconfiguration to adapt to the workload by adding or removing slots. Systems that adopt task-level deployment can decide how to allocate resources to individual tasks when they are activated, whereas systems that adopt job-level deployment need to suspend and resume the entire job, which increases the overhead for performing a reconfiguration. The mechanisms that dynamically modify the resources (slots) available to a DPS can be activated either manually or by an automated service that monitors the utilization of resources and implements the allocation and deallocation policies. All commercial systems implement automated reconfiguration, frequently by relying on external platforms for containerization, such as Kubernetes, or for cluster resources management, such as YARN. The only exceptions for which we could not find official support for automated reconfiguration are Storm [94] and Kafka Streams [18].

Systems that belong to this class deploy tasks on activation, when their input data becomes available. Tasks store intermediate results on a persistent data bus, which allows to selectively restart them in the case of failure. This approach is best suited for long running batch jobs and was pioneered in the MapReduce batch processing system [38]. It has been widely adopted in various extensions and generalizations. HaLoop optimizes iterative computations by caching loop-invariant data and by co-locating tasks that reuse the same data across iterations [21]. Dryad [51] generalizes the programming model to express arbitrary dataflow plans and enables developers to flexibly select the concrete channels (data bus in our model) that implement the communication between tasks. CIEL [72] extends the dataflow model of Dryad by allowing tasks to create other tasks dynamically, based on the results of their computation. Spark is the most popular system of this class [99]: it inherits the dataflow model of Dryad and supports iterative execution and data caching like HaLoop. Spark Streaming [98] implements streaming computations on top of Spark by splitting the input stream into small batches and by running the same job for each batch. It implements task state using native Spark features: the state of a task after a given invocation is implicitly stored as a special data item that the task receives as input in the subsequent invocation.

In systems with task-level deployment, job compilation considers dynamic information to create tasks. For instance, the number of tasks instantiated to perform a data-parallel operation depends on how the input data is partitioned. Similarly, the deployment phase uses dynamic information to submit tasks to workers running as close as possible to the their input data. Hadoop (the open source implementation of MapReduce) and Spark adopt a delay scheduling [97]. They put jobs (and their tasks) in a FIFO queue. When slots become available, the first task in the queue is selected: if the slot is located near to the input data for the task, then the task is immediately deployed, and otherwise each task can be postponed for some time to wait for available slots closer to their input data. Task-level deployment enables sharing of compute resources with other applications: in fact, most of the systems that use this approach can be integrated with cluster management systems.

Task-level deployment also influences how systems implement fault tolerance and ensure exactly once delivery of results. Batch processing systems simply re-execute the tasks involved in the failure. In absence of state, the results of a task depend only on input data and can be recomputed at need. Intermediate results may be persisted on durable storage and retrieved in the case of a failure or recomputed from the original input data. Spark Streaming [98] adopts the same fault tolerance mechanism for streaming computations. It segments a stream into a sequence of so-called micro-batches and executes them in order. Task state is treated as a special form of data; it is periodically persisted to durable storage and is retrieved in the case of a failure. Failure recovery may require activating failed tasks more than once, to recompute the task state from the last state persisted before the failure.

Dynamic reconfiguration is available in all systems that adopt task-level deployment. Systems that do not provide any state abstraction can simply exploit new slots to schedule tasks when they become available and remove workers when idle. In the presence of task state, migrating a task involves migrating its state across activations: as in the case of fault tolerance, this is done by storing task state on persistent storage.

In the case of job-level deployment, all tasks of a job are deployed onto the slots of the computing infrastructure on job registration. As a result, this class of systems is better suited for streaming computations that require low latency: indeed, no scheduling decision is taken at runtime and tasks are always ready to receive and process new data. Storm [94] and its successor Heron [56] are stream processing systems developed at Twitter. They offer a lower-level programming API than dataflow systems discussed previously, asking developers to fully implement the logic of each processing step using a standard programming language. Flink [24] is a unified execution engine for batch and stream processing. In terms of a programming model, it strongly resembles Spark, with a core API to explicitly define job plans as a dataflow of functional operators, and domain-specific libraries for structural (relational) data, graph processing, and machine learning. One notable difference involves iterative computations: Flink supports them with native operators (within jobs) rather than controlling them from the driver program. Timely dataflow [71] offers a lower-level and more general dataflow model than Flink, where jobs are expressed as a graph of (data-parallel) operators and data elements carry a logical timestamp that tracks global progress. Management of timestamps is explicit, and developers control how operators handle and propagate them, which enables various execution strategies. For instance, developers may choose to complete a given computation step before letting the subsequent one start (mimicking a batch processing strategy as implemented in MapReduce or Spark), or they may allow overlapping of steps (as it happens in Storm or Flink). The flexibility of the model allows for complex workflows, including streaming computations with nested iterations, which are hard or even impossible to express in other systems. The preceding systems rely on direct and ephemeral channels (typically, TCP connections) to implement the data bus. Kafka Streams [18] and Samza [74], instead, build a dataflow processing layer on top of Kafka [55] durable channels. In systems that adopt job-level deployment, job compilation and deployment only depend on static information about the computing infrastructure. For instance, the number of tasks for data-parallel operations only depends on the total number of slots made available in workers. As a result, this class of systems does not support sharing resources with other applications: all resources need to be acquired at job compilation, which prevents scheduling decisions across applications at runtime.

We observed three approaches to implement fault tolerance and delivery guarantees. First, systems such as Flink and MillWheel [7] periodically take a consistent snapshot of the state. The command to initiate a snapshot starts from sources and completes when it reaches the sinks. In the case of failure, the last completed snapshot is restored and sources replay data that was produced after that snapshot, in the original order. If sinks can detect and discard duplicate results, this approach guarantees exactly once delivery. Second, Storm acknowledges each data element delivered between two tasks: developers decide whether to use acknowledgements (and retransmit data if an acknowledgement is lost), providing at least once delivery, or not, providing at most once delivery. Third, Kafka Streams relies on the persistency of the data bus (Kafka): it stores the task state in special Kafka topics and relies on two-phase commit to ensure that upon activation a task consumes its input, updates its state, and produces results for downstream tasks atomically. In the case of failure, a task can resume from the input elements that were not successfully processed, providing exactly once delivery (unless data elements are received out of order, in which case it retracts and updates previous results leading to at least once delivery). These three mechanisms are also used for dynamic reconfiguration, as they allow a system to resume processing after a new deployment.

Early dataflow systems were not well suited for iterative computations, which are common in graph processing algorithms. To overcome this limitation, an alternative computational model was developed for graph processing, known as vertex-centric [68]. In this model, pioneered by the Google Pregel system [66], jobs are iterative: developers provide a single function that encodes the behavior of each vertex \(v\) at each iteration. The function takes in input the current (local) state of \(v\) and the set of messages produced for \(v\) during the previous iteration; it outputs the new state of \(v\) and a set of messages to be delivered to connected vertices, which will be evaluated during the next iteration. The job terminates when vertices do not produce any message at a given iteration. Vertices are partitioned across workers and each task is responsible for a given partition. Jobs are continuous, as tasks are activated multiple times (once for each iteration) and store the vertex state across activations (in their task state). Tasks only communicate by exchanging data (messages between vertices) over the data bus, which is implemented as direct channels. One worker acts as a leader and is responsible for coordinating the iterations within the job and for detecting possible failures of other workers. Workers persist their state (task state and input messages) at each iteration: in the case of a failure, the computation restarts from the last completed iteration. Several systems inherit and improve the original Pregel model in various ways:

(i) by using a persistent data bus, where vertices can pull data when executed, to reduce the overhead for broadcasting state updates to many neighbor vertices [62]; (ii) by decoupling communication and processing in each superstep, to combine messagess and reduce the communication costs [45]; (iii) by allowing asynchronous execution of supersteps, to reduce synchronization overhead and inactive time [62]; (iv) by optimizing the allocation of vertices to tasks based on topological information, to reduce the communication overhead; (v) by dynamically migrating vertices between tasks (dynamic reconfiguration) across iterations, to keep the load balanced or to place frequently communicating vertices on the same worker [30]; and (vi) by offering sub-graph centric abstractions, suitable to express graph mining problems that aim to find sub-graphs with given characteristics [91].

For the sake of space, we do not discuss all systems derived from Pregel here, but the interested reader can refer to the detailed survey by McCune et al [68].

DMSs are designed to execute lightweight jobs that read and modify a shared state. We identified a few systems that support some form of heavy-weight job.

Several works aim to integrate data management and processing within a unified solution. S-Store [27] integrates stream processing capabilities within a transactional database. It uses an in-memory store to implement the shared state (visible to all tasks), the task state (visible only to individual tasks of stream processing jobs), and the data bus (where data flowing from task to task of stream processing jobs is temporarily stored). S-Store uses the same concepts as VoltDB [89] to offer transactional guarantees with low overhead. Data management and stream processing tasks are scheduled on the same engine in an order that preserves transactional semantics and is consistent with the dataflow. S-Store unifies input data (for streaming jobs) and invocations (of data management jobs, in the form of stored procedures): this is in line with the conceptual view we provide in Section 2.

SnappyData [70] has a similar goal to S-Store but a different programming and execution model. It builds on Spark and Spark Streaming, and augments them with the ability to access a key-value store (shared state) during their execution. In the attempt to efficiently support heterogeneous types of jobs, SnappyData lets developers select how to organize the shared state, such as in terms or format (row oriented or column oriented), partitioning, and replication. It supports group atomicity and isolation using two-phase commit and multi-version concurrency control, and integrates fault detection and recovery mechanisms for Spark tasks and their effects on the shared state.

StreamDB [31] and TSpoon [5] take the opposite approach with respect to S-Store by integrating data management capabilities within a stream processor. StreamDB models database queries as stream processing jobs that receive updates from external sources and output new results to sinks. Stream processing tasks can read and modify portions of a shared state: all database queries that need to access a given portion will include the task responsible for that portion. StreamDB ensures group atomicity and isolation without explicit locks: invocation of jobs are timestamped when first received by the system, and each worker executes tasks from different jobs in timestamp order. TSpoon does not provide a shared state but enriches the dataflow model with

(i) the ability to read (query) task state on demand and (ii) transactional guarantees in the access to task state.

Developers can identify portions of the dataflow graph (denoted as transactional sub-graphs) that need to be read and modified with group atomicity and isolation. TSpoon implements atomicity and isolation by decorating the dataflow graph with additional operators that act as transaction managers. It supports different levels of isolation (from read committed to serializable) with different tradeoffs between guarantees and overhead.

Hologres [53] is used within Alibaba to execute both analytical jobs and interactive jobs. The system is designed to support high volume ingestion data from external sources and continuous jobs that derive information to be stored in the shared state or to be presented to external sinks. The shared state is partitioned across workers. A worker stores an in-memory representation of the partition it is responsible for and delegates durability to an external storage service. The distinctive features of the system are

(i) a structured data model where state is represented as tables that can be physically stored row-wise or column-wise depending on the access pattern, and (ii) a scheduling mechanism where tasks are deployed and executed onto workers based on load balancing and prioritization of jobs that require low latency.

Dynamo [39] is a NoSQL key-value store used by Amazon to save the state of its services. State elements are arbitrary values (typically, binary objects) identified by a unique key. Jobs consist of operations on individual state elements: retrieve the value associated with a key (get) or insert/update a value with a given key (put). Dynamo builds a distributed hash table: workers have an associated unique identifier, which determines the portion of shared state (set of keys) they are responsible for. Shared state is replicated, so multiple workers are responsible for the same key. When a worker receives a client request for a key, it forwards the request to one of the workers responsible for that key. Clients may also be aware of the distribution of shared state portions across workers and use this information to better route their requests. Dynamo uses a quorum-based approach where read and write operations on a key need to be processed by a quorum of the replicas responsible for that key. Users can set the number of replicas for each key, the read quorum, and the write quorum to trade consistency and durability for performance and availability. After a write quorum is reached, updates are asynchronously propagated to remaining replicas. In the case of transient unavailability of one replica, another node can temporarily store writes on its behalf, which avoids blocking write operations while preserving the desired degree of replication for durability. When Dynamo is configured to trade consistency for availability, replicas may diverge due to concurrent writes: to resolve conflicts, Dynamo adopts a versioning system, where multiple versions of a given key may exist at different replicas. Upon write, clients specify which version they want to overwrite: Dynamo uses this information to trace causality between updates and automatically resolves conflicts when it can determine a unique causal order of updates. In the case of concurrent updates, Dynamo preserves conflicting versions and presents them to clients (upon a read) for semantic reconciliation. Dynamic reconfiguration is a key feature of Dynamo. Nodes can be added and removed dynamically, and shared state portions automatically migrate. Dynamo adopts a distributed (gossip based) failure detection model, whereas failure recovery takes place by simply redistributing state portions over remaining nodes.

DynamoDB. DynamoDB² is an evolution of Dynamo that aims to simplify operational concerns: it is fully managed and offered as a service. It exposes the same data model as Dynamo but uses a single-leader replication protocol, where all writes for a key are handled by the leader for that key, which propagates them synchronously to one replica (for durability) and asynchronously to another replica. Read operations can be either strongly or weakly consistent: the former are always processed by the leader, whereas the latter may be processed by any replica, even if the replica lags behind of updates. Replicas can be located in different data centers to support wide-area deployments. DynamoDB stores data on disk using B-trees and buffers incoming write requests in a WAL. It supports secondary indexes that are asynchronously updated from the log upon write. Recent versions of DynamoDB enable automated propagation of changes to external sinks. They also offer an API to group individual operations in transactions that provide group atomicity and configurable group isolation. Transactions can be configured to be idempotent, thus offering exactly once delivery. To detect failures, the leader of each partition periodically sends heartbeats to all replicas. After some heartbeats are lost, the remaining nodes use the Paxos consensus algorithm to elect a new leader. As an additional fault tolerance mechanism, B-trees and logs are periodically checkpointed to durable storage. Amazon offers automatic scaling of DynamoDB as a service: users can select the expected read and write throughput for a given table (blocks of key-value pairs), and DynamoDB automatically increases the amount of resources dedicated to that table to meet the requirements.

Redis. Redis [64] is a single-node in-memory key-value store. Since version 3.0, Redis Cluster provides a distributed implementation of the store. In terms of a data model, Redis differs from other key-value stores in that it provides typed values and optimized operations for those types. For instance, a value may be declared as a list, which supports appending new elements without overwriting the entire list. Redis provides a scripting language to express driver programs (stored procedures) that run system-side. Redis supports data dispatching to sinks in the form of a publish-subscribe service: clients can subscribe to a given key and be notified about the changes to that key. Users can configure their desired level of durability (for fault tolerance), with options ranging from no persistence to using periodic checkpointing to CL. Redis Cluster partitions data by key and uses single-leader asynchronous replication. Alternative solutions are available for wide-area deployments, where redirecting all writes to a single leader may be unfeasible. For instance, Redis CRDTs offer multi-leader replication with automated conflict resolution based on conflict-free replicated data types. Dynamic reconfiguration with migration of shared state portions is supported but not automated.

Other Key-Value Stores. Several other stores implement the key-value model with design and implementation strategies that are similar to those presented earlier. For the sake of space, we only discuss their key distinguishing factors. Voldemort³ and Riak KV⁴ follow the same design as Dynamo. Like Redis, Aerospike [84] adopts single-leader replication within a single data center and multi-leader replication for wide-area deployments. It focuses on both horizontal scalability (with workers on multiple nodes) and vertical scalability (with multiple workers on the same node). It adopts a hybrid storage model where key indexes are kept in memory but concrete values can be persisted on disk, and a threading model that reduces locking and contention by assigning independent shared state portions to threads. PNUTS [33] provides single-leader replication in wide-area deployments: it uses an intermediate router component to dispatch jobs invocations to responsible workers, and relies on an external messaging system to implement the data bus that propagates updates to replicas. The same service can also notify external sinks. Thanks to their simple interface, key-value stores are sometimes used as building blocks in distributed software architectures: Memcached is used as a memory-based distributed cache to reduce data access latency at Facebook [73]. RocksDB⁵ is used to persist task state in the Flink stream processing system (also discussed later). Other data stores offering a key-value model are presented in the following as part of NewSQL databases.

BigTable [29] and its open source implementation HBase⁶ use a wide-column data model: shared state is organized in tables, where each row is associated with a fixed number of column families. A column family typically contains multiple columns that are frequently accessed together. Tables associate a value (binary object) to a row and a column (within a column family), and they are range partitioned across workers by row and physically stored (compressed) per column family. Jobs can read and update individual values and perform table scans. Rows are units of isolation: accesses to columns of the same row from different jobs are serialized, and this is the only task grouping guarantee that BigTable offers. BigTable adopts a leader-worker deployment, where a leader is responsible for assigning shared state portions to workers. Initially, each table is associated with a single worker, but it is automatically split when it increases in size. Clients retrieve and cache information about state distribution and submit their requests (tasks) involving a given state portion to the worker responsible for that state portion. BigTable can store multiple versions for each value. Versions are also visible to clients, which can retrieve old versions and control deletion policies. Writes always append new versions of a value, which improves write throughput to support frequent updates. Workers store recent versions in memory in LSM trees and use an external storage service (the GFS distributed filesystem) for durability. Background compaction procedures prune old versions from memory and from the storage. BigTable uses replication only for fault tolerance. Specifically, it relies on the replication of the GFS storage layer, where it also saves a command log. In the case of failure of a worker, the latest snapshot of its shared state portion is restored from GFS and from the commands in the log that were not part of the snapshot. BigTable supports wide-area deployments by fully replicating the data store in each data center. Replicas in other data centers may be only used for fault tolerance or they can serve client invocations, in which case they provide eventual consistency. Similar to key-value stores, BigTable provides dynamic reconfiguration by migrating state across available workers.

Cassandra. Cassandra [57] combines the wide-column data model of BigTable and the distributed architecture of Dynamo. Like BigTable, Cassandra uses LSM trees with versioning and background compaction tasks to improve the performance of write-intensive workloads. It offers a richer job definition language that includes pre-defined and user-defined types and operations. It supports task grouping only for compare-and-swap operations within a single partition. Like Dynamo, it uses a distributed hash table to associate keys to workers, and provides replication both in cluster and wide-area deployments, using a quorum-based approach for consistency. The quorum protocol can be configured to trade consistency and durability for performance, setting the number of local replicas (within a data center) and global replicas (in the case of wide-area deployments) that need to receive and approve a task before the system returns to the client. In the case of weak (eventual) consistency, Cassandra uses an anti-entropy protocol to periodically and asynchronously keep replicas up-to-date, using automated conflict resolution (last write wins).

MongoDB [32] is representative of document stores, an extension of key-value stores where values are semi-structured documents, such as binary JSON in the case of MongoDB. MongoDB jobs can insert, update, and delete entire documents but also scan, retrieve, and update individual values within documents. Recent versions of MongoDB support simple data analytic jobs expressed as pipelines of data transformations. External systems can register as sinks to be notified about changes to documents. Shared state partitioning can be either hash-based or range-based. Clients are oblivious of the location of state portions and interact with the data store through a special worker component that acts as a router. Shared state portions can be replicated for fault tolerance only or also to serve read queries. Replication is implemented using a single-leader protocol with semi-synchronous propagation of changes, where clients can configure the number of replicas that need to synchronously receive an update, thus trading durability and consistency for availability and response time. By default, only single-document jobs are atomic. Recent versions also support distributed transactions using two-phase commit for atomicity and multi-version concurrency control for snapshot isolation.

CouchDB. CouchDB [10] adopts the same document data model as MongoDB. Early versions only support complete replication of shared state, allowing clients to read and write from any replica to improve availability. Replicas are periodically synchronized and conflicts are handled by storing multiple versions of conflicting documents or fields, delegating resolution to users. Since version 2.0, CouchDB provides a cluster mode with support for shared state partitioning and quorum-based replication, where users can configure the number of replicas for shared state portions, and quorum for read and write operations, thus balancing availability and consistency. CouchDB is conceived for Web applications and provides a synchronous HTTP API for job invocation. It makes the list of changes (change feed) to a document available for external components, which can consume it either in pull mode or in push mode (thus representing the sink components in out model). CouchDB lets users define multiple views for each document. In our model, we can see views as the results of registered jobs that are triggered by changes to documents. Computations that create views are restricted to execute on individual documents, but their results can then be aggregated by key (mimicking the MapReduce programming model). CouchDB supports dynamic reconfiguration with addition and removal of nodes, and redistribution of shared state portions across nodes. However, reconfiguration is a manual procedure.

AsterixDB. AsterixDB⁷ is a semi-structured (document) store born as a research project that integrates ideas from NoSQL databases and distributed processing platforms. AsterixDB offers an SQL-like declarative language that integrates operators for individual documents as well as for multiple document (like joins, group by). Jobs are converted into a dataflow format and run on the Hyracks data-parallel platform [19]. Interestingly, the platform deploys jobs task by task but does not rely on a persistent data bus. Rather, when all input data for a task become ready, it dynamically establishes network connections from upstream tasks that deliver the results of their computation before terminating. AsterixDB supports querying shared state as well as external data from active and passive sources. Like BigTable and Cassandra, it stores state in LSM trees, which improve the performance of write operations [9]. It supports partitioning but does not currently support replication. As part of its shared state, AsterixDB can store indexes to simplify accessing external data. It offers isolation only for operations on individual values, using locking to update indexes. Its data structure offer fault tolerance through logging, but it does not currently implement fault detection nor dynamic reconfiguration mechanisms.

InfluxDB⁸ is a DMS for time-series data. Shared state is organized into measurements, which are sparse tables resembling wide columns in BigTable and Cassandra. Each row maps a point in time (primary key) to the values of one or more columns. Developers need to explicitly state which columns are indexed and which are not, thus balancing read and write latency. InfluxDB uses a storage model (time structured merge trees) that derives from LSM trees. It stores measurements column-wide and integrates disk-based storage and an in-memory write cache to improve write performance. Jobs are defined in the InfluxQL declarative language and are restricted to single measurements. InfluxDB supports active sources, and mimics continuous jobs through periodic execution of one-shot jobs, which write their results inside the database and/or send them to sinks. InfluxDB partitions and replicates shared state across workers. Write operations are propagated semi-synchronously across replica workers for fault tolerance. Read operations can access any of the replica workers that contain the requested data, leading to weak consistency. InfluxDB adopts a leader-worker approach, where leader nodes store meta-data about membership, data partitioning, continuous queries, and access rights, and worker nodes store the actual data. Leader nodes are replicated with strong consistency for fault tolerance, using the Raft consensus protocol. InfluxDB offers dynamic reconfiguration to migrate data and add new workers, but the process is manual.

Gorilla. Gorilla [75] is an in-memory time-series store that Facebook uses as a cache to an HBase data store. HBase stores historical data compressed with a coarser time granularity, whereas Gorilla persists the most recent data (26 hours) in memory. Gorilla uses a simple data model where values for each measure always consist of a 64-bit timestamp and a 64-bit floating point number. It uses an encoding scheme based on bit difference that reduces the data size by 12 times on average. Jobs in Gorilla only perform simple read, write, and scan operations. A few ad hoc jobs have been implemented to support correlation between time series and in-memory aggregation, which is used to compress old data before writing it to HBase. Gorilla supports geo-replication for disaster recovery but trades durability for availability. Written data is asynchronously replicated, and it can be lost before it is made persistent. Gorilla supports dynamic adaptation by redistributing the key space across workers.

Monarch. Monarch [3] is a geo-distributed in-memory time-series store designed for monitoring large-scale systems and used within Google. Its data model stores time-series data as schematized tables. Each table consists of multiple key columns that form the time-series key, and a value column, which stores one value for each point in the history of the time series. Key columns include a target field, which is the entity that generates the time series, and a metrics field, which represents the aspect being measured. Monarch has a hierarchical architecture. Data is stored in the zone (data center) in which it is generated and sharded (by key ranges, lexicographically) across nodes called leaves. Data is stored in main memory and asynchronously persisted to logs on disk, trading durability to reduce write delay. Monarch offers a declarative, SQL-like language to express jobs, which can be either one-shot or continuous. In the latter case, they are evaluated periodically and store their results in new derived tables (materialized views). Jobs are evaluated hierarchically: nodes are organized in three layers (global, zone level, leaves), and the job plan pushes tasks as close as possible to the data they need to consume. Each level also stores an approximate view (compressed index) of what the nodes in the lower-level store. This enables optimizing communication across levels by avoiding pushing tasks to nodes that have no data related to that task. Monarch supports dynamic reconfiguration: it monitors lower-level nodes and redistributes data (key ranges) across nodes to adapt to changes in the load.

Peregreen. The design of the Peregreen [96] time-series database aims to satisfy the following requirements. First, cloud deployment of large volumes of historical data: as the scale of data is prohibitive for in-memory solutions, Peregreen relies on storage services such as distributed filesystems or block storage. It limits a raw data footprint by supporting only numeric values and by representing them in a compressed columnar format: each column is split into chunks and data in each chunk is represented using differences between adjacent values (delta encoding), further compressed to form a single binary array. Second, fast retrieval of data through indexing: Peregreen uses a three-tier data indexing, where each tier pre-computes aggregated statistics (minimum, maximum, average, etc.) for the data it references. This allows to quickly identify chunks of data that satisfy some conditions based on the pre-computed statistics and to minimize the number of interactions with the storage layer. Peregreen jobs can only insert, update, delete, and retrieve data elements. Retrieval supports limited conditional search (only based on pre-computed statistics) and data transformation. Chunks are versioned, and a new version is created in the case of modifications. Peregreen is designed for cluster deployments and uses the Raft algorithm to reach consensus on the workers available at any point in time and the state portions (indexes to the storage layer) they are responsible for. This enables dynamic adaptation, with addition and removal of workers for load balancing and elasticity. State portions are replicated at multiple workers for fault tolerance.

TAO [20] is a data store that Facebook developed to manage its social graph, which contains billions of entities (e.g., people and locations) and relations between them (e.g., friendship). TAO offers a simple data model where entities and relations have a type and may contain data in the form of key-value pairs. It provides a restricted API for job definition, to create, delete, and modify entities and relations, and to query relations for a given entity. With respect to other graph data stores, it does not support queries that search for sub-graphs that satisfy specific constraints (path queries or sub-graph pattern queries). TAO is designed to optimize the latency of read jobs, as it needs to handle a large number of simultaneous user-specific queries. To do so, TAO implements shared state in two layers: a persistent storage layer based on a relational database (MySQL) and a key-value in-memory cache based on memcache. TAO is designed for wide-area deployments. Within a single data center, both the persistent layer and the cache are partitioned. The cache is also replicated using a single-leader approach: clients always interact with follower cache servers, which reply to read operations in the case of cache hit and propagate read operations in the case of cache miss as well as write operations to the leader cache server, which is responsible for interacting with the storage layer and for propagating changes. Across data centers, the storage layer is fully replicated with a single-leader approach: reads are served using the data center local cache or storage layer, so they do not incur latency, whereas all write operations are propagated to the leader data center for the storage layer. Data is replicated between the storage layer and the cache as well as across data centers asynchronously, which provides weak consistency and durability. Dynamic reconfiguration is possible in the case of failures: in the case in which a leader cache or storage server fails, replicas automatically elect a new leader.

Unicorn. Unicorn [35] is an indexing system used at Facebook to search its social graph. The shared state of Unicorn consists of inverted indexes that enable retrieving graph entities (vertices) based on their relations (edges) and the data associated with them. For instance, in a social graph, one could use an index on relations to retrieve all people (vertices) that are in a friend relation with a given person, or a string prefix index to retrieve all people with a name that starts with a given prefix. Unicorn is optimized for read-only queries (jobs), in the form of index lookups and set operations (union, intersection, difference) on lookup results. Jobs are evaluated by exploiting a hierarchical organization of workers into three layers:

(i) index servers store the shared state (the indexes) partitioned by results, meaning that any index server will store a subset of the results for each index lookup; (ii) a rack aggregator per rack is responsible for merging the (partial) query results coming from individual index servers; and (iii) a top aggregator is responsible for merging the (partial) query results coming from each rack.

Interestingly, Unicorn jobs can dynamically start new tasks: this feature is implemented as an apply function that performs new lookups starting from the results obtained in previous ones. This feature can be used to implement iterative computations—for instance, to find the friends of friends of a given person, Unicorn can first retrieve the direct friends and then lookup for all their friends, using result set of the first lookup (direct friends) as a parameter for the second lookup. Unicorn indexes are kept up-to-date with the content of the social graph using a periodic procedure that runs on an external computation engine. Unicorn tolerates failures through replication. However, it is possible for some shared state portions to remain temporarily unavailable: this may result in incomplete results when searching (if some index serves do not reply), which is acceptable in its specific application domain.

Deuteronomy [59] is a data store designed for wide-area deployments that decouples storage functionalities from transactional execution of jobs. In Deuteronomy, the driver program runs client-side and submits jobs (queries) to a transaction component. The component ensures group atomicity and isolation for all read and write operations performed on the shared state, using a locking protocol. The actual storage is implemented in a separate layer. Deuteronomy supports any distributed storage component that offers read and write operations for individual elements (e.g., a key-value store), and is oblivious of the actual location of the workers implementing the storage component, which may be geographically distributed. Deuteronomy provides fault tolerance through logging and replication, and enables dynamic reconfiguration by independently scaling both the transactional and the storage component.

FoundationDB. FoundationDB [100] is a transactional key-value store, which aims to offer the core building blocks (hence the name) to build scalable distributed DMSs with heterogeneous requirements. The use of key-value abstractions provides flexibility in the data model, on top of which developers can build various types of abstractions. For instance, each element may encode a relation indexed by its primary key. Jobs consist of a group of read and write requests issued by a driver program that runs client-side. Like Deuteronomy, FoundationDB is organized into layers, each of them offering one of the core functionalities of a transactional DMS and each implemented within a different set of workers, thus enabling independent scaling. A storage layer persists data and serves read and write requests. A log layer manages a WAL. A transaction system handles isolation and atomicity for multiple read and write requests. Transactional semantics is enforced by assigning timestamps to operations and by checking for conflicts between concurrent transactions after they have been executed: in the case of conflicts, the transaction aborts and the driver program is notified. Fault tolerance is implemented by replicating both the log layer (synchronously) and the storage layer (asynchronously). Replication of the storage layer is also used to serve read requests in parallel. Moving data between workers that implement the storage layer and the log layer is also used when adding or removing workers for scalability (dynamic reconfiguration).

Solar. Like Deuteronomy and FoundationDB, Solar [101] is a transactional key-value store that decouples storage of shared state from transaction processing. The transaction layer uses a timestamp-based optimistic concurrency control and stores a WAL in main memory. The storage layer persists checkpoints of the shared state. Together, they form an LSM tree. The key distinguishing feature of Solar is that the transaction layer is implemented as a centralized service, replicated for fault tolerance of the WAL.

Spanner [34] is a semi-relational database: shared state is organized into tables, where each table has an ordered set of one or more primary key columns and defines a mapping from these key columns to non key columns. Spanner provides transactional semantics and replication with strong consistency for cluster and wide-area deployments. At its core, Spanner uses standard database techniques: two-phase locking for isolation, two-phase commit for atomicity, and synchronous replication of jobs results using Paxos state machine replication. Jobs are globally ordered using timestamps, and workers store multiple versions of each state element (multi-version concurrency control). This way, read-only jobs can access a consistent snapshot of the shared state and do not conflict with read-write jobs. A consistent snapshot preserves causality, meaning that if a job reads a version of a state element (cause) and subsequently updates another state element (effect), the snapshot cannot contain the effect without the cause. Traditional databases ensure causality by acquiring read and write locks to prevent concurrent accesses, but this could be too expensive in a distributed environment. The key distinguishing idea of Spanner is to serve a consistent snapshot to read-only jobs without locking. To do so, it uses an abstraction called TrueTime, which returns real (wall-clock) time within a known precision bound using a combination of GPS and atomic clocks. Tasks of read-write jobs first obtain the locks for all the data portions they access: a coordinator for the job assigns that job with a timestamp at the end of its time uncertainty range, then it waits until this timestamp is passed for all workers in the system, releases the locks, and writes the results (commits). The waiting time ensures that jobs with later timestamps read all writes of jobs with earlier timestamps without explicit locking. Using TrueTime, Spanner also supports consistent reconfiguration—for instance, to change the database schema or to move data for load balancing. More recently, Spanner has been extended with support for distributed SQL query execution [14].

CockroachDB. CockroachDB [90] is a relational database for wide-area deployments. It shares many similarities with Spanner and integrates storage and processing capabilities within each node. As a storage layer, it relies on RocksDB, a disk-based key-value store that organizes data in LSM trees. It replicates data across nodes, ensuring strong consistency through Raft consensus. On top of this, it partitions data with transactional semantics: it uses an isolation mechanism based on hybrid physical and logical clocks (similar to Spanner) but integrates it with an optimistic protocol that, in the case of conflicts, attempts to modify the timestamp of a job to a valid one rather than re-executing the entire job. CockroachDB compiles SQL queries into a plan of tasks that can be either fully executed on a single worker or in a distributed dataflow fashion. Interestingly, CockroachDB also enables users to configure data placement across data centers. For instance, a table can be partitioned across a Region column to ensure that all data about one region is stored within a single data center. This may improve access time from local client and enforce privacy regulations.

Deterministic Execution Calvin. Calvin [92] is a job scheduling and replication layer to provide transactional semantics and replication consistency on top of non-transactional distributed data stores such as Dynamo, Cassandra, and MongoDB. It is currently implemented within the Fauna database.⁹ The core Calvin layer is actually agnostic with respect to the specific data and query model. In fact, Fauna supports document and graph-based models in addition to the relational model. Calvin builds on the assumption that jobs are deterministic. Its core idea is to avoids as much as possible expensive coordination during job execution by defining a global order for tasks before the actual execution. In Calvin, workers are organized in regions: each region contains a single copy of the entire shared state, and each worker in a region is fully replicated in every other region. All workers that contain a replica of the same state portion in different regions are referred to as a replication group. Invocations from clients are organized in batches: all workers in a replication group receive a copy of the batch, and they coordinate to agree on a global order of execution. Under the assumption of deterministic jobs, this approach ensures consistent state across all replicas in a replication group. Replicas are used both to improve access for read-only jobs, which can be executed on any replica, and for fault tolerance, as in the case of a failure remaining replicas can continue to operate without interruption. Invocations are stored in a durable log, and individual workers can be resumed from a state snapshot by reapplying all invocations that occurred after that snapshot. Calvin supports different replication protocols with different tradeoffs between job response time and complexity in fail over. Deterministic jobs executed in the same order lead to the same results in all (non-failing) replicas. Specifically, they either commit or abort in all (non-failing) replicas. Accordingly, under the assumption that at least one replica for each shared state portion does not fail, Calvin can provide atomicity without expensive protocols such as the classic two-phase commit: if some tasks may abort due to violation of integrity constrains, they simply inform other tasks of the same transaction with a single (one-phase) communication. Global execution order is also used for isolation: Calvin exploits a locking mechanism where tasks acquire locks on their shared state portion in the agreed order. This requires knowing up front the exact state portions accessed within each job: when they cannot be statically determined (e.g., due to state-dependent control flow), Calvin runs reconnaissance jobs that perform all read accesses to determine the state portions of interest. However, during the actual execution, shared state may have changed, and the real jobs may deviate from reconnaissance jobs and try to access different portions, in which case they are deterministically restarted. Interestingly, Calvin provides the same strong semantics both for cluster and for wide-area deployments. The initial coordination increases the latency to schedule a batch of jobs, affecting the response time of individual jobs in wide-area deployments, but the batching mechanisms can preserve throughput.

Explicit Partitioning and Replication Strategies VoltDB. VoltDB [89] is an in-memory relational database developed from the HStore research project [88]. In VoltDB, clients register stored procedures, which are driver programs written in Java and executed system-side. They include multiple jobs, are compiled on registration, and are executed on invocation. Jobs can also write data to sinks. The key idea of VoltDB is to let users control database partitioning and replication, so they can optimize most frequently executed jobs. In particular, VoltDB preserves the same (transactional) execution semantics as centralized databases while minimizing the overhead of concurrency control. By default, all tasks that derive from a single driver program represent a transaction and are guaranteed to execute with group atomicity and isolation. In the worst case, this is achieved through blocking coordination (two-phase commit for atomicity and timestamp-based concurrency control for isolation). However, VoltDB avoids coordination for specific types of jobs by exploiting user-provided data about data partitioning and replication. Users can specify that a relational table is partitioned based on the value of a column. For instance, a Customer table may be partitioned by region, meaning that all customers that belong to the same region (have the same value for the attribute region) are stored in the same shared state portion. Jobs that only refer to a given region can then be executed by the single worker responsible for that state portion, sequentially, without incurring expensive concurrency control overhead. Every table that is not partitioned is replicated in every worker, which optimizes read access from any worker at the cost of replicating state changes. Users need to select the best partitioning and replication schema to improve performance for the most frequent jobs. VoltDB also supports replicating tables (including partitioned ones) for fault tolerance. Replicas are kept up-to-date by propagating and executing tasks to all replicas, under the assumption that tasks are deterministic. VoltDB is designed for cluster deployment as clock synchronization and low latency are necessary to guarantee that timestamp-based concurrency control and replication work well in practice. However, VoltDB also provides a hybrid deployment model where a database is fully replicated at multiple geographical regions. These replicas can be used only as an additional form of fault tolerance (with asynchronous propagation of state) or can serve local clients. In this case, consistency across regions is not guaranteed, and conflicts get resolved using pre-defined automatic rules. VoltDB supports manual reconfiguration of both partitioning and workers but requires stopping and restarting the system.

Primary-Based Protocols Aurora. Aurora [95] is a relational database offered as a service by Amazon. Aurora builds on two key design choices:

(i) decouple the storage layer from the query processing layer and (ii) store the log of changes (WAL) in the storage layer instead of the actual shared state. Shared state is materialized only to improve read performance, and materialization can be performed asynchronously without increasing write latency.

The storage layer—that is, the write log—is replicated both to improve read performance and for fault tolerance. To ensure consistency, read and write operations use a quorum-based approach. The processing layer accepts jobs from clients in the form of SQL queries, which are always executed within a single worker. Specifically, to guarantee isolation, Aurora assumes that a single worker is responsible for processing all read-write jobs at any given point in time. Read-only jobs can be executed on any worker, which can read a consistent snapshot of the state without conflicting with concurrent writes. In Aurora, the storage and processing layers can scale independently: the processing layer is stateless, whereas the storage layer only needs to replicate the log.

Socrates. Socrates [11] is a relational database offered as a service in the Azure cloud platform (under the name SQL DB Hyperscale).¹⁰ Like Aurora, Socrates decomposes the functionality of a DMS and implements them as independent services. Its design goals include quick recovery from failure and fast reconfiguration. To do so, it relies on four layers, each implemented as a service that can scale out when needed. First, compute nodes handle jobs, including protocols for group atomicity and isolation. There is one primary compute node that processes read-write jobs and an arbitrary number of secondary nodes that handle read-only jobs and may become primary in the case of failure. Compute nodes cache shared state pages in main memory and on SSD. Second, a log service logs write requests with low latency. Third, a storage service periodically applies writes from the log to store the shared state durably. Fourth, a backup service stores copies of the storage layer for fault tolerance.

Tango [16] is a service for storing metadata. Application code (the driver program) executes client-side and reads and accesses a shared state consisting of Tango objects. Clients store their view of Tango objects locally in-memory, and this view is kept up-to-date with respect to a distributed (partitioned) and durable (replicated) totally ordered log of updates. Tango objects can contain references to other Tango objects, thus enabling the definition of complex linked data structures such as trees or graphs. Although the log is physically partitioned across multiple computers, all operations are globally ordered through sequence numbers, which are obtained through a centralized sequencer service: the authors demonstrate that this service does not become a bottleneck for the system when serving hundreds of thousands of requests per second. Clients check if views are up-to-date before performing updates, thus ensuring totally ordered, linearizable updates. Tango also guarantees group atomicity and isolation using the log for optimistic concurrency control.

A1 [22] is an in-memory database that resembles TAO and Trinity in terms of data model (typed graphs) and jobs (changes to graph entities and graph pattern matching queries). The distinguishing characteristic of A1 is that it builds on a distributed shared memory abstraction that uses RMDA implemented within network interface cards [41]. A1 stores all the elements of the graph in a key-value store. Jobs may traverse the graph and read and modify its associated data during execution. A1 provides strong consistency, atomicity, and isolation using timestamp-based concurrency control. Fault tolerance is implemented using synchronous replication in memory and asynchronous replication to disk. In the case of failures, users can decide whether to recover the last available state or only state that is guaranteed to be transactionally consistent.

SDG [43] is a programming model that extracts a dataflow graph of computation from imperative code (Java programs). In SDG, developers write driver programs that include mutable state and methods to access and modify it. Code annotations are used to specify state access patterns within methods. The program executor (a client-side compiler) analyzes the program to extract state elements and task elements, representing shared state and data-parallel tasks in our model. If possible, state elements are partitioned across workers. Similarly, task elements are converted into multiple concrete tasks, each accessing one single state element from the shared state portion of the worker it is deployed on. For instance, consider a program including a matrix of numbers and two methods to update a value in the matrix and to return the sum of a row. The matrix would be converted into a state element partitioned by row (since both methods can work on individual rows). Each method would be converted into a data-parallel task, with one instance of the task per matrix partition. State elements that cannot be partitioned are replicated in each worker, and the programming model supports user-defined functions to merge changes applied to different replicas. In terms of execution, SDGs are similar to stream processing systems such as Storm or Flink: jobs are continuous, and tasks communicate through direct TCP channels. SDGs rely on periodic snapshots and re-execution for fault tolerance: the same mechanism is adopted to dynamically scale in and out individual task elements depending on the input load they receive [26].

TensorFlow. TensorFlow [1] is a system for large-scale machine learning that extends the dataflow model with explicit shared mutable state. Jobs represent machine learning models and include operations (data transformations) and variables (shared mutable state elements representing the parameters of the machine learning model). Frequently, jobs are iterative and update variables at each iteration. The specific application scenario does not require strong consistency guarantees for accessing shared state, so tasks are allowed to execute and read/write variables asynchronously. If needed, TensorFlow permits some form of barrier synchronization—for instance, to guarantee that all tasks perform an iteration step using a given value of variables before they get updated with the results of that step. TensorFlow tasks can be executed on heterogeneous devices (e.g., hardware accelerators), and users can express explicit placement constraints. Since version 2, part of the dataflow plan may be defined at runtime, meaning that tasks can dynamically define and spawn downstream tasks based on the input data. Variables can be periodically checkpointed to durable storage for fault tolerance. In the case of failure, workers can be restarted and they restore the latest checkpointing available, with no further consistency guarantees. TensorFlow has been conceived from the very beginning as a distributed platform, but many other libraries for machine learning, initially designed for a single machine, inherited its stateful dataflow execution model: the most prominent and most widely adopted example is PyTorch.¹⁴

Tangram. Tangram [50] is a data processing framework that extends the dataflow model with explicit shared mutable state. It implements task-based deployment but allows tasks to access and update an in-memory key-value store as part of their execution, which enables optimizing algorithms that benefit from fine-grained updates of intermediate states of computations (e.g., iterative algorithms or graph processing algorithms). By analyzing the execution plan, Tangram can understand which parts of the computation depend on mutable state and which parts do not, and optimizes fault tolerance for the job at hand. Immutable data is recomputed using the same (lineage) approach of MapReduce and Spark, thus re-executing only tasks that are necessary to rebuild the data. Mutable state is periodically checkpointed. In general, Tangram does not provide group atomicity or isolation for state: simultaneous accesses to the same state portions from multiple tasks may be executed in any order.

ReactDB [79] extends the actor-based programming model with database concepts such as relational tables, declarative queries, and transactional execution semantics. ReactDB builds on the abstraction of reactors, which are logical actors that embed state in the form of relational tables. Each reactor can query its internal state using a declarative language (SQL) or can explicitly invoke other reactors. Invocations across reactors are asynchronous and retain transactional semantics: clients invoke a root reactor and all invocations it makes belong to the same root-level transaction, and are atomic and isolated. The core idea of ReactDB is that developers control data partitioning across reactors and distributed execution: on one extreme, they can place all state in a single reactor and mimic a centralized database; on the other extreme, they can assign a single table (or a single partition of a table) to each reactor, which maximizes distributed execution. With this model, the execution plan is partly implicit (within a single reactor) and partly explicit (calls between reactors). ReactDB is currently a research prototype. As such, it lacks a distributed implementation, replication, fault tolerance, and dynamic reconfiguration mechanisms.