IoTRepair: Flexible Fault Handling in Diverse IoT Deployments

IoT systems integrate physical processes with digital connectivity. These systems perform simple tasks such as motion-activated light switches, as well as complex tasks such as controlling traffic lights in a smart city. Regardless of purpose and complexity, IoT systems often use an edge device as a centralized gateway that connects devices in physical environments and use a cloud back-end to synchronize device states and provide interfaces to control and monitor devices.

Faults in IoT systems can occur due to flaws in components of devices, the cloud, and communication between them. While faults may happen in any of these components, device failures are more common due to factors such as minimal computational resources, energy constraints, architectural problems, improperly configured systems, and disruptive environmental conditions [17, 18, 24, 27, 31]. For instance, devices in smart homes could experience faults on an average of two hours a day due to power loss, network disruption, and hardware failure [13]. In a similar vein, experiments in exposed environments recorded half of the devices reporting incorrect states due to severe weather conditions [36].

An IoT system often consists of five components (Figure 1, right): (i) a set of sensors and actuators such as the temperature sensor, light sensor, door lock, thermostat, and leak sensor shown in the figure; (ii) an auxiliary Micro-Controller Unit (MCU) to read the raw sensor values; (iii) an edge device with low-power CPU cores and controllers for communicating with sensors, actuators, and the cloud; (iv) a cloud provider for performing advanced computation and interacting with the user; and (v) either a battery or a power supply unit for each sensor/actuator and the edge device to power each of these components. Components (ii) to (v) are often housed in an edge device such as a hub. A fault in any of these components can lead to errors in the system. We will use a temp-control app that subscribes to a thermostat, a heater, and window actuators to illustrate the different fault types and their consequences (see Figure 1). The app opens the windows and notifies the user through SMS when the temperature exceeds a user-defined threshold and closes the windows and sets the heater to a user-defined temperature value when the temperature is below the threshold.

We divide faults into two categories, extending the terminology in previous fault tolerance 1 work [6] (see Table 1). Visualizations of the various fault types and the effects they can have on devices are shown in Figure 2. Fail-stop faults happen when a device stops functioning and is unresponsive to external requests; for example, when a device loses power, network communication fails; as another example, a software or hardware error halts device operation. A fail-stop fault in the thermostat would cause our example app to halt entirely, and such a fault in the window or the heater would remove the functionality of those actuators.

Non-fail-stop faults relate to a response by the device that diverges from the desired device state. These faults can manifest in a variety of ways with different effects, as shown in Reference [24]: an outlier fault causes a device to report an incorrect state for a very brief period, usually the duration of just 1 poll. For example, the temperature reported by a temperature sensor may fall outside of a reasonable range for a single device poll. An unhandled outlier fault in the thermostat could cause the app to send an unintended notification that incorrectly indicates the temperature is above the threshold. A stuck-at fault happens when a device cannot change state, maintaining the same state despite changes in environment or actuation commands. An example is if the app opens the window, but the temperature sensor fails to decrease the temperature as it should, which causes a safety issue because an open window enables a burglar to break in the house. Conversely, in commercial buildings with automated doors, a door that is stuck-at closed could cut off access to areas of the building or even trap individuals in certain areas. A stuck-at fault could also cause energy data analytics to be incorrect and home temperature to surge if the heater gets stuck with the always-on state. A high variance fault appears when a device’s state fluctuates back and forth more rapidly than the environment dictates. For example, the temperature value fluctuates between high and low more than the environment temperature. If this variance crosses the user-defined temperature threshold, then this could cause several issues by repeatedly opening then closing the window and turning the heater on and off. A spike fault happens when there is a rapid increase or decrease in the device state; e.g., the temperature decreases faster than the actual temperature in the environment. This could cause safety issues by incorrectly closing the window and turning on the heater, overheating the house—wasting energy and causing discomfort, or possibly even damage if there are temperature-sensitive objects in the environment or the temperature was already very hot. In an industrial setting, a downward spike fault in a temperature sensor for a furnace could cause it to heat to a degree far greater than intended, damaging what is inside and possibly the furnace itself. This could also cause faults in other devices as a now improperly created product is processed as though it were correctly created.

The existing fault tolerance methods discussed in Section 1 and later in this section cannot address this wide array of fault types. Those methods either address only a few specific faults, rely on extensive replication, or cannot handle the effects of cascading behavior.

To evaluate over every possible permutation of these faults would create an impossibly large test size. Therefore, our evaluation in Section 6 targeted only the variables that were consistent across most faults, meaning fault duration and false state, leaving specific elements such as partial fail-stop or the spike change rate as constants. We argue this is acceptable, as these are unlikely to affect the latency of IoTRepair, and have a minor effect on the power consumption.

We also divide faults based on what has caused the device to begin faulty behavior. These classifications are important, because they can influence how effective a fault-handling function is, as well as how long it takes to complete. Our fault causes and how they affect system behavior differently are defined below.

A power failure occurs when a device no longer has sufficient power to operate, either due to an expended battery or loss of power supply. It completely stops the operation of the device; further, it cannot be repaired by any remote means; so any fault handling other than replication or rollback wastes time and energy.

A communication disruption occurs when the hub is either unable to communicate with the device or the communication is unreliable. In the case where communication is impossible, this cause is no different than a power failure, but if communication is unreliable, then normal operation can be approximated by retrying polls and actuations, and it may be possible to repair, as the device can still receive function commands.

A software error occurs when a bug in the device’s software causes one of the fault types to occur. This can only occur in devices with sufficient software to host bugs, such as a Thermostat, and can most often be repaired by re-initializing the device’s software or by power-recycling the device.

A hardware error occurs when a defect in the device’s hardware causes one of the fault types to occur. This can occur in any device and may be fixable by power-cycling the device one or more times; otherwise, it is no different from a power failure regarding remote fault handling.

As can be seen in the definitions, the cause of a fault mostly affects what form of fault handling may be able to correct the device behavior, if any. Other causes, such as environmental conditions or adversarial attacks, do not add anything significant to this list of causes from IoTRepair’s perspective.

As discussed in the introduction, as IoT systems are primarily autonomous, users generally have little interaction and oversight. Therefore, it is highly desirable to have a largely automated IoT fault identification and handling system.

Fault tolerance in IoT platforms. To understand how IoT platforms identify and handle faults, we have studied eight major IoT platforms to characterize their fault identification and handling methods. The results for each platform are shown in Table 2, with undetected meaning that the fault is not recognized by the platform, silent meaning that the fault is recognized but the system does not inform apps of the fault, general error meaning that an error is thrown but the error does not contain the fault type, and detailed error meaning an error is thrown and it specifies the fault type. For instance, SmartThings provides a developer API to obtain the current device state and can notify an app when the device state is changed [34]; an app can check for the state changes before continuing execution. This requires additional developer effort, and if a developer neglects this check, an app would operate ignorant of whether a device failure exists. Additionally, if a failed device is queried, then SmartThings returns the last known device state or some default value, which may potentially return stale sensor states when a device is unreachable. This enables an app to continue functioning, yet the stale states may inadvertently actuate incorrect device states.

The results in the table show that only Android Things gives enough information to handle fail-stop faults effectively, and no platforms give the means for applications to handle non-fail-stop faults. In contrast, IoTRepair is able to handle these types of faults automatically and further provides a toolset to allow developers on any platform to perform their own handling.

Research Literature on IoT Fault Identification and Handling. Fault identification techniques aim to determine the presence of a fault and determine the faulty device and type. We characterize the fault identification techniques for IoT systems into three groups. First, network traffic-based techniques analyze sensor data packets to detect faults [28, 44]. Network-level techniques are effective at identifying fail-stop failures but ignore non-fail-stop faults. Second, redundant sensor-based techniques use data from multiple homogeneous sensors and exploit the fact that spatial sensing of close sensors should yield similar sensor states [5, 10, 31]. However, these systems are costly due to the requirement of multiple sensors and, more importantly, ineffective when considering that simultaneous sensor faults are common in IoT. In the last group, data obtained from different types of sensors are used for fault identification. These techniques apply to environments such as smart homes where various sensors states are available and often correlated [17, 19, 23]. While these techniques differ in scope, precision, and methodology, they do not identify fault types.

Very little research has been performed in IoT fault handling. The explored techniques often aim at specific environments and faults. A fault-tolerant technique [39] for redundancy-free UAV sensors uses imperfect replication to allow near-correct device operation in the presence of a sensor failure. While this work poses a good argument for the value of imperfect replication, the functions are specific to UAV sensors and contribute little to repairing faults. Rivulet [3] aims at removing an edge device as a single point of failure to mitigate connection faults by distributing communication to devices. The core drawback of this technique is that it does not handle the far more common sensor and actuator faults. Transactuations is a fault-handling technique introduced in Reference [30] to address some flaws that transactions have in IoT environments. This enhanced technique better prevents physical device states from losing synchronization with software variables. While this is useful for preserving dependencies, it does not repair faults and cannot correct errors that occur before the fault is detected. To the best of our knowledge, no current techniques develop a fault handler for diverse IoT device failures and bundle fault identification and fault handling to mitigate faults. CEFIoT [15] and its extension IoTEF [16] use a fault tolerance model designed for distributed systems with the goal of ensuring data processing and delivery. While it is very effective at increasing the resilience of systems with advanced distributed computation, it assumes the collected data is correct at the source, which may not be true in the case of a faulty device, nor the cascading effects that can cause. The system also does not attempt to repair faulty devices actions or consider the changing state of an IoT environment.

Challenges in IoT Fault Handling. Compared to the fault handling in other computing platforms such as distributed systems and cloud services, fault handling in IoT systems raises a set of unique challenges. We present these challenges below and discuss briefly how we address each.

First, there is no single fault handler technique that can address all possible device faults, due to the vast array of heterogeneous IoT devices and the diverse set of environments in which they are deployed. Implementing an optimal handling technique for a specific fault is highly contextual—one cannot define the impact or correctness of fault handling without understanding the environment. Due to the heterogeneity of IoT devices, various devices often require different fault-handling techniques executed to fit power, environmental, and computation constraints. IoTRepair addresses this challenge by introducing a set of functions that may be invoked in a flexible order with custom parameters through device-driven defined schemes. These schemes can be chosen to meet the system and environmental requirements such as safety, security, and availability. We present the fault-handling functions and schemes in Section 4.

Second, optimal handling of a specific fault is highly contextual due to varying user and developer demands and environmental factors. For instance, some users may prefer energy conservation over real-time fault handling, and a developer may desire their apps suspended when a fault occurs. IoTRepair enables users and developers to define their requirements in a configuration file, which is updated by runtime environmental input.

Third, due to dynamic application code across deployments, faulty behavior can cascade throughout the environment unpredictably. This cascading behavior causes a faulty device in an app to incorrectly trigger an event in another app. To illustrate, we consider a secure-home app co-located with the temp-control app. The secure-home app controls a presence sensor, a door lock, and a window actuator. The app keeps home secure—door-locked and window-closed—when the user is not at home and notifies the user if the house becomes insecure. An unhandled spike fault in the thermostat controlled by the temp-control app causes an increase in temperature when the user is not at home and causes the window to open. The secure-home app then sends an SMS alert to the user saying that the house is unsafe—making the user become unnecessarily anxious. This behavior is correct from the secure-home app, as the window should not be open; however, the window is opened because of a fault that triggers the event handler of the temp-control app. This example shows that faults of a faulty device can propagate beyond the device through influencing the environment and interactions between apps. This requires fault tolerance that can either prevent or remedy these behaviors.

Finally, a fault-handling mechanism at installation and runtime must require minimum user interaction and domain expertise. For instance, a solution that only notifies users about the fault may lead to safety issues, as the users might not be available, and the fault may require a real-time response. IoTRepair provides automated fault-handling functionality with user configurations.

Device-based functions are an implementation of isolated fault handling, which considers only the state of the faulty device, but not the overall state of the system. The effectiveness of such a function depends only on the specifications of the faulty device and the nature of the fault. These functions do not directly affect and are not affected by any other devices in the system.

Activate Redundant Devices. Functional replication is a well-studied technique in many domains, with applications in cloud environments, distributed systems, and even some IoT implementations. We argue that IoT devices can provide replication in two distinct ways: (a) a duplicate of the primary device; (b) a device that is not a complete duplicate but provides similar capabilities; for example, a camera that is located in the same room as a fault motion sensor can detect motion to replicate the functionality of the motion sensor. IoTRepair currently can only automatically implement replication of type (a), but type (b) is a quite reasonable future addition, as the feasibility was demonstrated in earlier work [39].

The user can use the configuration file to specify any known replicating devices of type (a). IoTRepair’s API function ActivateRedundantDevice allows the system to continue to run unaffected by a detected fault, so long as the configuration file specifies the replicating device ID if one exists. If there is, then it redirects polls and actuation commands from the faulty device to the replicating device. Using this method, IoTRepair does not rely on native replication support from the platform. This also places all replication logic in the cloud, which keeps latency and overhead to a minimum.

To ease the burden of writing replicating devices in the configuration file, IoTRepair’s runtime includes a method to automatically detect likely candidates for replication, similar to the fingerprinting method in Reference [12]. It examines the list of connected devices through platform capabilities for devices with matching type and automatically generates relevant configuration data. Finding a replicate device in an IoT environment must consider the fact that similar devices are not necessarily co-located. To address this, IoTRepair observes the sensor states during polling for a configured time and checks whether devices report the same states consistently. If the two devices’ states and transition timings remain within acceptable bounds, then these devices are identified to be replicating. Figure 4 illustrates how two co-located motion sensors would be detected as replicating, as their states match consistently. In contrast, an unrelated presence sensor and distant motion sensor are not treated as duplicates.

Retry Device State. Retry is effective at handling short, transient faults in devices, or faults that only incur a chance of failure. The first purpose of Retry is stalling device execution long enough to let the fault resolve itself and preventing excessive handling from being performed. This can resolve very short faults, such as outliers, that only take place over a few milliseconds. The second is that, if the device is an actuator or the proper state of a device is known, then through repeated polling and/or actuating the fault may be resolved. Retry takes parameters of the device ID and three optional arguments: a function pointer for a validation function, a list of expected device states, and whether the fault is fail-stop or not. When used by the automated handler, these may be supplied by the fault identification module. Additionally, each device is configured with a set number of retries. If a function pointer to a validation function is given, then the validation function will be called to check if the fault is resolved (until the retry limit is reached). If the expected device states are passed, then the retry will poll and/or actuate the device continually until an expected state is returned or retry limit is reached. If an expected value is returned, then it will mark the fault as resolved and end fault handling. If the fault is identified as fail-stop, then any successfully returned state will be treated as an expected value.

Software and Hardware Restart. Restarting can potentially correct both software faults, where the onboard software on the device encounters a bug, or hardware faults, where the hardware enters a bad state, so long as it can be remedied by a power-cycle. SoftwareRestart and HardwareRestart take a device ID and send a signal to the faulty device to initiate a software and hardware restart, respectively. The configuration file will also be read for the device to determine the max number of attempts to restart. If no acknowledgment signal is received, then the command will be repeated, and if it is not acknowledged a configured number of times in a row, then the restart will be aborted and return with an error code. If the restart is acknowledged, if the fault was not fail-stop, then the function will disable the app and device suppression for the passed device and then wait for a length of time equal to the upper bound on fault detection for the loaded fault identification module. If no fault is detected during this time, then the restart resolves the fault and fault handling ends. If a fault is detected or the fault was fail-stop, then the restart function ends unsuccessfully.

The implementation of restart functions is device-dependent. Many IoT devices such as Honeywell Z-Wave Thermostat [33] and open-source electronics platforms such as Arduino implement the device restart functionality.

User Notification. Notify takes a device ID and notifies users of the fault and the result of fault handling. The implementation of the notify call is platform-dependent; we inform users through alert messages on the console, yet text messages or push notifications are possible through IoT platform APIs. Event triggers for user notification can be configured for each device to notify the user when a fault cannot be repaired by the handler, when a fault has been repaired by the handler, when a fault occurs, or any combination of these.

Environment-based functions are an implementation of linked fault handling, which mitigates cascading behavior. It responds to a fault by considering the states of all devices in an IoT environment; e.g., linked fault handling should close the window that was incorrectly opened by an app due to the faulty thermostat. As environment-based functions can affect all device states in an environment, they are capable of mitigating cascading failures across the system.

Checkpoint. Checkpoint stores all device states in the system at the time it is called. In general, a checkpoint is a sequence of pairs of devices and device states. For example, in our Arduino deployment each device either has a digital or analog value for its current state. So, a system with a presence sensor, motion sensor, and light may have a Checkpoint that looks like [“Presence”: 1, “Motion”: 0, “Light”: 1], which means Presence is detected, Motion is detected, and Light is on. We divide device states into sensor states and actuator states. Sensor states are read-only states that collectively represent the state of the environment and drive behavior. Actuator states can be read and modified through actuation, and they are the states that can be rolled back.

We assume that there is a causal relation from the sensor to actuator states; in other words, an app follows the well-known sensor-computation-actuator structure: It takes the sensor states, performs some computation, and then performs actuation to modify the actuator states. Based on this, IoTRepair includes a novel, history-based checkpoint/rollback mechanism (1) that during checkpointing records the history of device states, and (2) that during rolling back restores the most likely actuator states by looking up the history according to the current sensor states. We next discuss this mechanism in detail.

First, the automated fault handler invokes the checkpoint function after every actuation that does not trigger cascading actuations. This means a checkpoint is taken after actuations where no subscribing application initiates an actuation based on the new state. The handler determines these states by checking all apps logic subscribed to the actuated device to ensure no additional actuation is performed as a result.

Second, we must ensure no faults were present at the time a checkpoint was taken. We must wait until a period equal to the upper bound on the fault-detection time has passed without a fault being identified to verify this. If the fault-identification module does not have a known upper bound, then we wait for a user-defined length of time, with a default value of 30 minutes.

When enough time has passed, the new checkpoint is then stored in a history log, which holds checkpoints and their frequencies. If no previous checkpoint has matching sensor states, then a new checkpoint is appended to the log. This Checkpoint will be assigned a frequency for tracking how often it has occurred in the system, with an initial value of 1. This value acts as a tie-breaker when choosing between possible target Checkpoints. If the sensor states match an existing checkpoint’s sensor states, then the frequency is incremented if actuator states also match, or the actuator states are overwritten, and the frequency is reset to be one if the actuator states differ. Only storing the most recent actuator states for a set of sensor states keeps the log from exploding in size. For the same reason, checkpoints are removed from history if they remain unused for an extended time, as determined by the configuration. If computation and storage space is relatively high, such as on a cloud server, then we could store all possible actuator configurations for each sensor set. In this case, actuator states could also be used when deciding between Checkpoints, where the Checkpoint that would require less actuations to match is selected. This would allow for causal relationships between actuators to be included in rollback.

Figure 5 provides an example of how checkpoints are taken over time in a system as a result of changes in actuator states. The example uses a simple system with two devices: a motion sensor and a light actuator. The figure shows checkpoints being taken during a series of actuations, which occur at timestamps \(\mathtt {t_1}\) through \(\mathtt {t_4}\) , with initial time \(\mathtt {t_0}\) before actuations. On the left side of the figure, the state of the system after the actuation completes is shown; on the right is the list of checkpoints at each time, starting from an empty set. The first two checkpoints at times \(\mathtt {t_1}\) and \(\mathtt {t_2}\) are new states, since there are no existing checkpoints that match the sensor states. For these new checkpoints, the time and current device states are recorded, and the frequency is set to 1. The checkpoint taken at \(\mathtt {t_3}\) has the same device states as an existing checkpoint; so it updates the checkpoint timestamp and the frequency. At time \(\mathtt {t_4}\) , a checkpoint that matches the sensor states is taken, but the actuator states do not match those of an existing checkpoint; so the matching checkpoint’s timestamp is updated, actuator states are overwritten, and frequency is reset.

Rollback. Rollback performs a series of actuations to match the system state to the checkpoint that best reflects the current system sensor values. Algorithm 1 details the steps for computing a rollback that finds all potential matches, determines the best match, and performs actuation according to the best match.

We have designed three techniques for choosing the best checkpoint to target for rollback: most-recent, fail-norm, and fail-safe. The configuration for the faulty device determines which technique will be used. Lines 1 and 2 of the algorithm involve retrieving which of these to use based on the faulty device.

Most Recent targets the last checkpoint that occurred in the system. This is the most basic method and assumes that faults can be detected very quickly relative to the speed that the environment typically changes. This assumption means that most if not all of the recent changes in the environment should be due to the faulty behavior; so they can safely be rolled back to allow the system to continue from a recent safe point. A good example would be in an environment where detecting faults has a high priority, like a hospital. It also works well with complex devices that are more likely to be able to self-diagnose and report faults. Most recent is also good in environments where it is assumed many devices are likely to have faults at once. This is because other rollback methods decrease in accuracy as more faults occur and decrease what can be known about the environment state. An example environment where this would be the case is one that spreads across many networks, where one node going down may result in many devices suffering a network failure.

Fail-norm finds which checkpoint best matches the current known environment. First, only checkpoints where all sensor states match the current sensor states in the system are considered. As the states of faulty sensors cannot be trusted, their sensor states are not considered when determining if a checkpoint matches the current sensor states. Because of this, it is possible that multiple checkpoints match the current known state of non-faulty sensors. If there is more than one match, then the frequency of checkpoints is used as a tie-breaker. As previously mentioned, this method assumes a causal relationship from sensors to actuators, and also assumes a large majority of devices are non-faulty at rollback time. Fail-norm is best suited to devices whose states have a high degree of correlation with other devices. For example, a motion sensor’s state may be highly correlated with the state of a presence sensor, which allows the rollback to use the presence sensor to determine what the faulty motion sensor’s state should be. It is also better in environments without single points of failure, such as a factory with a backup generator for power failure.

Fail-safe uses the fail-safe configurations for all of the actuators in the system to reduce the list of checkpoints to only those where actuators match their fail-safe states; e.g., the fail-safe configuration for an alarm might be to have it on. Once the list has been reduced, a checkpoint will be selected in the same manner as fail-norm. This is the only method that will proceed with the rollback even if no valid checkpoint is found or one of the devices that must be changed is faulty. Even in these cases, the bare minimum is to put all devices with fail-safe states into those states. Similar to fail-norm, fail-safe assumes devices are correlated for selecting checkpoints, but more importantly assumes that there are devices that have known safe states that are critical for them to enter if their proper state becomes ambiguous. The best example for this is when the faulty device drives the behavior of many important devices. In a commercial building with automatic doors, a faulty fire alarm may mean that all interior doors must open to avoid possibly preventing individuals from escaping a possible fire. A similar case would be a smart home that closes and locks all entrances if the presence sensor becomes faulty to secure the home while the location of the resident is unknown.

Line 3 of the algorithm is where a rollback technique is used to iterate through the list of previously taken checkpoints to find the best match for the rollback type. Most Recent will choose the checkpoint with the latest timestamp. Fail-norm finds the checkpoint that matches all known current sensor states best, using how frequently it has occurred as a tie-breaker. Fail-safe is the same is fail-norm, but will choose only checkpoints where all devices with fail-safe states in the configuration file match those states. If there are no matches, then the rollback fails and terminates.

Rollbacks can be dangerous, as a partial rollback can result in an otherwise impossible transition and enter an invalid system state. For this reason, our rollback fails if any actuator that needs to be changed is currently faulty (lines 6–7). We note that this captures the heterogeneous nature of IoT devices compared to similar functions in distributed systems [21] and cyber-physical systems [20]. We further note that as IoTRepair assumes that the identification module does not provide exactly when the fault occurred in the device, all actuators that do not match the chosen checkpoint must be actuated. If the fault time were provided, then it would be possible to rollback only the effects of the fault; we discuss a possible extension in Section 7. We also implement a data rollback as part of the system rollback. Specifically, as long as the rollback does not fail, any faulty sensors are set to their values at the time of the checkpoint until the device is repaired or another rollback occurs (Line 8).

Figure 6 gives an example of how fail-norm rollback would operate in a system with two sensors, motion and presence, and a lock actuator for a door. It shows that IoTRepair’s rollback can mitigate dangerous faults that would otherwise persist until the user can repair the faulty device, as long as another sensor’s state is correlated with the faulty sensor’s state with high frequency. In the example of Figure 6, a presence sensor that is stuck at home could cause the door to remain unlocked indefinitely. Fortunately, the motion sensor’s state is largely correlated with the presence sensor’s state, because it is likely to detect user motion when the user is at home. As a result, IoTRepair’s rollback can then use the motion sensor’s state to correct the door to be locked and secure the home, even when the presence sensor is faulty.

Transaction. A transaction performs a series of actuations and ensures that the actuations are performed atomically as a group, either all successfully completing or none completing. Currently, the transaction function is not used by IotRepair’s automated handler, yet app developers can invoke it to cause a series of actuations to execute atomically when a partially executed series may leave the system in an unsafe state. For instance, a developer can issue a call “transaction([[Window, Closed], [Heater, On]])” to close the Windows and turn on the heater as a transaction. Transactions are implemented through a standard undo-log-based algorithm. It performs each of these actuations in the order given in the passed array while recording the original states in an undo log. If any actuation fails, then the transaction is aborted, all devices are reset to their previous states, and an error is returned.

We describe functions necessary for the operation of the fault handler. These functions are not directly used during fault handling but can be invoked by the automated handler or application developers to modify handler configuration.

Device Suppression. It is not a library function, but rather a capability injected into the code of an edge device (e.g., a hub). This is because it is required to interrupt polls and event hooks at the edge device. Through this capability, the edge device can terminate polling the device’s state and block actuation commands sent to the device. This prevents events from a faulty device triggering incorrect actuations to other devices through app code. Device suppression is effective in correcting non-fail-stop faults. For example, if the smoke detector experiences a high variance fault, then it may rapidly switch between smoke-detected and smoke-not-detected states. This would cause the alarm to turn on and off until the fault is resolved. With device suppression, the smoke detector can be suppressed once it is identified as faulty.

Application Suppression. Similar to device suppression, application suppression is also a capability injected into the edge device code. Application suppression can be enabled or disabled for each app based on the configuration file. When a device is identified as faulty, application suppression checks the list of apps subscribed to the device’s events. For every app that application suppression is enabled, all events that would be sent to the app and all commands generated by the app are suppressed. This halts the execution of the app, which is desirable when an app has functionality that may put the system into an unsafe state if the state of one of its dependent devices is unknown. For example, suppose there is an app that opens the window when the presence sensor indicates the user is home and the house is too hot; when the presence sensor is faulty, the window may open if the app is not suppressed.

Update Device/App Configuration. These functions allow IoTRepair to update the configuration file at runtime by passing a device or application name and a list of fields to update. If the passed arguments are valid for the given fields, then the values in the configuration file are updated. App developers can also call this function with the same parameters to customize the configuration file for their requirements.

We introduce a set of built-in schemes to automate the functions introduced above (see Table 4). The schemes control the execution order of the functions to address environmental requirements such as safety and security. We choose to use the device type as a primary determinant of which scheme to use, since IoT device parameters introduce several limitations, e.g., duration of restarts and the presence of replicated devices, which impacts the optimal function ordering. Our four schemes are not designed to address all possible deployments, and additional schemes can easily be created for different deployment requirements and when new handling functions are developed. IoTRepair updates selected schemes based on device behavior during fault handling. We describe the purpose of each scheme. The Conservative scheme fits in environments where there are no strict time requirements, and energy conservation is the primary goal. For this reason, the functions are ordered by increasing cost of power to the device. The Permanent-expected scheme aims at devices that are unlikely to experience transient faults. It is similar to Conservative, but it skips over Retry, as Retry can only resolve transient faults. For instance, this would be suitable for a temperature sensor deployed in a stable home environment, since it is capable of restarting and does not control time-sensitive operations. The Long-restart scheme is used in devices that have excessively long software and hardware restart times, such as security cameras and a smart refrigerator. For this reason, it deploys rollbacks before attempting any restarts to attempt to first correct the system state. The Time-sensitive scheme aims to return the system to the desired state as soon as possible, for instance, when an IoT system that is responsible for security is unresponsive. It is similar to Long Restart, but skips Retry, as Retry delays more effective functions. This scheme fits industrial or vehicle environments where system integrity is the top priority.

For evaluating the effectiveness of IoTRepair, we constructed a set of faulty scenarios to inject into our setup and test how effective IoTRepair is at handling each scenario. These scenarios and the results are displayed in Table 7. Section 1 presents scenarios where a single fault is injected: For instance, in scenario 1, a failure in a presence sensor causes the door lock to be unlocked incorrectly. Section 2 presents scenarios where multiple faults are injected and Section 3 presents scenarios where a single fault is injected but it causes cascading behavior. For each scenario, we list what faults are injected, what the effect of the fault in the system is, which IoTRepair functions should be used to handle the fault, what the defined desired behavior should be once IoTRepair is finished, and whether IoTrepair was effective at achieving the desired behavior. These scenarios are complete over fault types and applications they affect, though not every possible combination. The scenarios were selected to cover the breadth of fault effects IoTRepair could have to deal with in our setup.

For executions, the state of non-faulty sensors is hard-coded to follow the behavior necessary for the listed fault effects. Faults are injected immediately, but fault detection takes 10 seconds to allow the effects to complete before IoTRepair activates. We manually determine the desired behavior for each scenario and check the states of all devices compared to our desired behavior to evaluate if IoTRepair effectively handled the fault. Desired behavior is generally a return to the state pre-fault, but for some instances where faults are not removed or caused the environment state to change, the desired behavior is defined to follow the intent of the applications as best as possible.

As shown in the table, most faults are resolved successfully, but there are a few instances where IoTRepair’s success is conditional on some element. For instance, in scenarios 1, 3, and 13, the desired behavior occurs, but, since Fail-safe rollback is used, devices unrelated to the fault are also rolled back. An optimal fault handler would only rollback the devices associated with the given fault; so in scenario 1 the alarm would not be turned on. This behavior is acceptable, because Fail-safe by definition places these devices in a safe state that will cause little to no harm. Additionally, for devices that do not have a Fail-Safe state, Fail-Safe rollback has the same requirements as Fail-Norm.

For Fail-Norm rollback, we also rely on consistent correlation between sensor states. In scenario 11 the correctness of the rollback has requirements, namely, that the non-faulty information indicates the correct state of the faulty devices. In this case, recent timing information in checkpoints indicates that at this time of day the heater is likely to be on. This is acceptable behavior, as it fits into the intended effect of Fail-Norm rollback.

Cascading Behavior. Our applications are susceptible to cascading behavior—which are cases where a fault in one device causes incorrect states in a second device through inter-application interactions. Cascading behavior can be unpredictable, because the application that places a device in a faulty state might not directly utilize the value of the faulty device, and the application interaction may not even be intended. Row 16 of Table 7 shows an example of this. In this scenario, a temperature sensor becoming stuck at a low temperature causes the following sequence of events and actuations in App3, App7, and App11:

heater-off \(\xrightarrow {\text{temp < 60}}\) App3 : heater-on \(\xrightarrow {\text{heater-on}}\) App7 :close-window

In this example, App7 causes the window to close as a cascading result of App3’s action responding to a fault in the temperature sensor. The complicated inter-application relationship of cascading behavior lends additional value to environmental functions such as rollback. Even if another function corrects the fault, the cascading behavior may not be automatically removed by application code. In this example, while the heater turning on causes the window to close, there is no code to open the window when the heater is turned off; so the window would incorrectly remain closed even after the temperature sensor fault is removed. In this case, a correct rollback can put the system back into the correct state.

For Table 7’s cascading behavior scenarios, we consider fault-handling schemes and whether the order of fault-handling functions in a scheme affects the behavior of IoTRepair. The only function whose position in a scheme matters is rollback. This is because not knowing the state of the device is only relevant for rollback, as it may affect which checkpoint is selected. Therefore, if a previous fault-handling function corrects a faulty device, then rollback is more likely to select the correct checkpoint to remove any cascading behavior the fault had. In scenarios 16 and 20, rollback must be run second, because in these instances Fail-Norm chooses the incorrect Checkpoint as the true sensor state is unlikely given other state information. In other scenarios rollback selects the correct target whether the fault is present or not, as the true device state of the faulty device is likely given the other state information. This is acceptable as it mostly relies on the accuracy of rollback, which, as shown below in our discussion of cascading behavior, is very high. Other than the interaction between rollback and other fault-handling functions, the only other interaction is that software restart should never follow hardware restart, since hardware restart corrects anything software restart would.

To further understand the effectiveness of rollback, we discuss the pros and cons of the three types of checkpoint selection that allow rollback to handle different types of cascading behaviors based on desired behavior: most recent, fail-norm, fail-safe.

Most Recent is the best at resolving cascading behavior that is detected quickly and is reversible. Reversible in this case means the state changes caused by the cascade can be undone. For example, a window that was opened can be closed, but it is not possible to reverse making coffee. A good example of when Most Recent rollback would correct a cascade in our implementation is the sample given in the text above, where a fault in the Temperature Sensor closes the window; a Most Recent rollback would turn off the heater and open the window, correcting the issue.

Fail-Norm is the best rollback for when there are only a few faults in the system and there are no possibly highly dangerous states based on the faulty devices. For example, in Table 7 scenario 16, we see that turning on the heater can cause the window to close incorrectly. Here, fail-norm may find a checkpoint where non-faulty sensors indicate the heater should be off and the window should be open and correctly actuate the devices. However, if the temperature sensor also caused a dangerous action, such as unlocking the door, then it may be better to rely on fail-safe.

Fail-Safe is the best rollback when neither of the conditions for Most Recent or Fail-Norm is true. As this rollback attempts to move all devices to their “secure” state, it should avoid any unsafe states, to the maximum degree that can be achieved given the fault set. For example, if we were to consider the Smoke Sensor cascade in Table 7 scenario 18, then the state of the alarm could be potentially dangerous if there is a fire; so Fail-Safe would turn on the alarm to establish the safest state until the fault is properly resolved.

For Most Recent, the effectiveness and accuracy are determined by how quick the fault is detected; this is independent of IoTRepair, which relies on a separate fault-identification module. Fail-Safe can be statically evaluated, given a set of faulty sensors the target rollback is known; so the only determinant is if it is possible. For example, we know fail-safe will always attempt to close the window; but if the window becomes stuck-at open, then there is nothing fail-safe can do to correct the state.

The only rollback that must be evaluated empirically during out evaluation is fail-norm, as its effectiveness and accuracy depends on how well the checkpoints IoTRepair collects represent system states. To this end, we evaluate Fail-Norm by running the program with 0 faults for 1,000 polls to get a baseline and gather checkpoints. We then run again with rollback enabled and attempt a rollback at each poll timestamp. For each rollback, we analyze if the correct checkpoint is chosen across every possible device being faulty, and if it is, then we mark that as accurate and if not, then it is marked as incorrect. We indicate a correct checkpoint by the checkpoint selecting the actuator target states that match the baseline at this timestamp. The primary cause of an incorrect checkpoint is when a sensor is faulty; it cannot be considered when selecting the best fit checkpoint; thus, two or more checkpoints may be “valid.” In this case, fail-norm selects the one that occurred the most during checkpointing. Within this experiment, only one rollback was incorrect by these standards, giving a 99.9% accuracy, at least in a system that stays stable for long periods of time and only experiences a single fault.

Overall, this evaluation shows that IoTRepair can effectively handle a wide range of faults, most of the time returning the system to normal operation or at the very least making the system behavior closer to intended behavior.

In this experiment, we evaluate the latency of functions in IoTRepair’s fault-handling library. Across all possible fault injection parameters there is a huge variety of scenarios and because of this large state space it would be time-consuming to experimentally cover the entire state space to measure the latency of each fault-handling function for each possible scenario. Therefore, for our evaluation, we decided to analyze a select subset of those to cover typical kinds of devices and typical faults that can happen to those devices. Among the 17 devices in our IoT testbed, we selected two sensors and two actuators. The two sensors include a simple digital sensor (a motion sensor) and an analog sensor (a temperature sensor). The two actuators include one with high frequency of actuation (a light actuator) and one with low frequency of actuation (a heater). These four devices are representative of the devices in our testbed. For example, a motion sensor and a contact senor are nearly identical in their characteristics. We also performed experimental analysis on devices not selected for evaluation and confirmed that there is little variance between the data for the representative devices and the data for the rest.

For each selected device, we picked the most typical kind of fault to inject for that device; we will explain this selection and its rationale soon. To get timing information, we record the total time between when a fault is injected and when a fault is either resolved by a fault-handling function or the function determines it cannot resolve the fault. The latency is calculated from an average of three executions for each scenario and each possible set of parameters. We note some parameter combinations are invalid; specifically, since power and communication disruption can only cause fail-stop faults, there is no need to run executions with a fault delay greater than or equal to fault duration, and only communication disruption has a communication probability parameter. Furthermore, when getting the timing information of a fault-handling function, we remove the communication time spent between the core machine, representing the cloud, and the Arduino board, representing the Hub. In our experiments, our communication time was on average 1.23 seconds, which is similar to cloud-based IoT environments [22], and significantly longer than localized implementations, such as mobile devices [8]. This communication time is removed because it is independent of the execution time of our IoTRepair functions, which is the main objective for this evaluation.

The evaluation results are presented in Figure 8 to Figure 11. For these figures, we label fault-handling functions by Success/Fail labels. A label specifies whether a fault-handling function completes its goal or not. For different functions, goal completion can mean different things; for restart, the goal is to remove the fault, while rollback and retry simply aim to mitigate the faulty behavior. Furthermore, even though a fault may have multiple parameters, through our experimental data, we found most parameters did not cause variation on the latency of fault-handling functions. Therefore, each figure will use only the most dominant parameter (in terms of its effect on fault-handling latency) as the x-axis.

Figure 8 presents the results of a Communication Disruption fault injected into a Motion sensor. We selected the Communication Disruption for two reasons. First, the motion sensor (located on a porch or at the front door) is the most likely to be far from the Hub, and therefore more likely to have periods of unreliable connection to the Hub. Second, since most motion sensors are battery-powered and simple, they are relatively unlikely to suffer other types of faults. The x-axis in this graph is the Communication Disruption Probability, which represents how likely the fault is to cause any given communication to fail. Each failed message causes additional time where the action must be repeated until completion or reaching the limit of attempts. For this reason, of all of our parameters this has the most impact for most functions, with the exception of rollback. Because rollback does not consider faulty devices when selecting the best checkpoint, the communication failure probability of a device has no impact on latency. Additionally, this Communication Disruption Probability parameter causes the highest standard deviation among sampled results, because each execution can vary wildly in latency based on how many times the action fails. But from this graph, we can see that even with an extremely high (80%) chance of failed communication, the latency only doubles compared to the case of having 30% chance to fail and remains under 60 milliseconds.

Figure 9 shows a power failure injected into a heater actuator. We selected power failure because it is the most likely fault for a heater to experience; anytime a home loses power or the power flickers, the heater can lose power—or if its connection to an outlet is disrupted in any way. The x-axis shows the amount of fault delay (i.e., the amount of time the fault-detection module detects the fault); even though the fault delay causes the largest change among applicable parameters, it still causes only a fraction of a percentage change in latency. With 25 ms as the highest latency, it is clear that latency is low for power failures in this evaluation; however, it is worth noting that only rollback can succeed in case of a power failure.

Figure 10 shows a hardware fault injected into a light actuator. This fault was selected because among all our devices lights have the shortest lifespan, generally burning out or otherwise failing far before other devices would suffer major hardware faults. The x-axis is selected to be duration, as it has the highest impact; though similar to the previous figure, the change is small in magnitude. The only exception is for failed rollbacks. This is because rollback attempts to actuate light to a certain state, and because it is faulty this cannot succeed; however, for each run the selected checkpoint and system state cause high variance in how many actuations are attempted. This is more impactful than successful rollbacks, since the devices that did succeed need to be reverted to the pre-rollback state. Despite this variance, the latency still remains below a fast 40 ms, on average.

Finally, Figure 11 presents a software fault injected into the temperature sensor. Of our deployed devices, only Temperature Sensor and Smoke Sensor are complex enough to suffer proper software faults and allow a software restart, and the temperature sensor is the more complex of two. Fault duration is the parameter with the most impact; so it is selected as the x-axis. But just like the previous two graphs, the impact on latency is still small. The average latency for all functions remains small, even with the relative complexity of temperature sensors.

It can be seen across these graphs that only the communication disruption probability makes any real impact on latency. For delay and duration, changes across the axis are small and do not exhibit any trend. For fault duration, this is because the typical duration is much longer than any element of IoTRepair latency. For fault delay, when the time spent on fault identification is removed, most IoTRepair functions are not sensitive to the state change over that time. The only function that can be is rollback, and in our experiments, the delay did not drastically change how many devices needed actuated. Also, besides the latency of these functions, transactions incur an average of 44.82-millisecond latency per-transaction. This is because the transaction must complete and verify each actuation before moving on to the next and record the previous states of devices in case a transaction rollback becomes necessary. We separate transaction and measure it independently, as unlike the other functions it operates whether a fault is present or not. This means it imposes a constant latency overhead on the system that must be accepted by the user, and that it would be pointless comparing it to functions that only have to be called once per fault. Finally, we also tracked the latency of checkpointing and replication. We chose not to include checkpointing in the graph in part because it occurs constantly whether a fault is present or not, and because both checkpoints and replication take fractions of milliseconds to finish. This is due to the lack of reliance on the Arduino Hub, with all calculation taking place on the far-faster machine representing the cloud implementation. Additionally, only motion and light devices have replicating devices in our setup.

Based on our evaluation, we argue that the amount of latency in IoTRepair is acceptable compared to the current state of IoT. First, the current state of the art of IoT fault identification takes far longer to detect most faults than our system takes to handle them. This is shown in more detail in our Related Work section; one such example is DICE [6], one of the fastest identification methods, and it still takes at least 10 minutes to identify faults. Compared to this, the latency of IoTRepair is insignificant. This cannot be said for fail-stop faults, though, which some systems may detect near-instantly, or at least within a matter of seconds, depending on the polling/event generation setup. Even in these cases, we argue our latency is acceptable, as the average fault duration is also quite high relative to the latency. As seen in Reference [13], faults can last hours, in which case, while handling them as fast as possible is still desirable, IoTRepair is fast enough to handle the vast majority of the harm the fault will cause. In the case of outlier faults, such as in Reference [36], the fault will be gone before IoTRepair can handle it, but IoTRepair can still rollback and likely fix any changes that the brief fault caused.

Comparing IoTRepair’s latency to other systems’ faces several challenges. First, most existing systems incur some constant latency cost to normal system operation rather than a time-to-fix latency like IoTRepair. Additionally, many papers do not report latency in a way conducive to comparisons. Many papers either omit latency information, evaluate some other metric such as throughput, or do not provide a baseline for comparison. Despite this, with some justifications, we can compare to several systems that show IoTRepair has acceptable latency. Transactuations [30] provide the execution time of 10 SmartThings apps with and without their system. They observe an average of 50% overhead when Transactuations are implemented. IoTRepair’s transactions incur slightly less overhead, with an overhead of 37.5% when compared to normal actuations. The comparison is not perfect, since Transactuations focus on preserving the synchronization of soft and hard states, while our transaction avoids unsafe state transitions; however, the implementation from a user’s standpoint is similar. Another system [1] implements several blockchain consensus algorithms to increase fault tolerance when some devices may provide incorrect information. It measures the latency of sending packages of 360 bytes, though it does not provide a ground-truth for when there is no blockchain implementation. Depending on the algorithm, number of nodes, and propagation model, the latency ranges from around 3 milliseconds to almost 500 milliseconds. So, depending on the circumstances, the latency may be better or worse than IoTRepair functions, but even in the best case where the overhead is only a few milliseconds this method imposes a constant cost. This cost is incurred far more frequently than our Transaction or Checkpoint and will slow the system far more than IoTRepair. Another method for implementing fault tolerance is to design an IoT architecture from the ground up to be Fault Tolerance, such as IoTEF [16]. As IoTEF is an independent architecture, it has no baseline to compare to. However, it does provide the latency of processing images. When sending images from the camera to the edge device it takes roughly 10–60 ms, depending on the image size. The median is similar to each of our fault-handling functions, showing that IoTRepair’s fault-handling functions have comparable latency to normal operating processes in IoTEF. Finally, Wang et al. [41] propose a system that adaptively determines whether edge devices need to replicate data in case of data loss. In their evaluation, the system imposed only an overhead of a couple milliseconds at worst from a retransmission-only baseline. This is less overhead than IoTRepair functions besides Replicate and Checkpoint, but once again it occurs far more frequently and targets only preventing data-loss due to a fault. Overall, IoTRepair imposes very little latency overhead during normal runtime when compared to other fault tolerance solutions, and the time to handle faults is relatively low.

In this evaluation, we measure the power overhead caused by IoTRepair and show that it is low cost in terms of power consumption compared to normal system operations. For this, we first tracked how much power was consumed across each execution and compared the power overhead of our fault-handling functions. We performed this using a Monsoon Power Monitor to measure the power consumed by the Arduino board during execution. We show these results with the same scenarios used in Section 6.2. The power consumption is shown as the average power consumption used during IoTRepair function calls.

Figure 12 presents the average power consumption of a fault-handling function for the scenario when a communication disruption fault is injected into a motion sensor. Most functions consume less than 200 milliwatts, which is an insignificant amount when compared to the fact that the average power consumed per poll cycle is 4.28 watts. A poll cycle represents the time from when a poll begins to when the next poll begins. The time it takes a cycle to complete affects how much power is consumed, since idle power consumption is relatively high for the system. Time is primarily consumed by communication time in the system, as discussed in Section 6.2. A non-faulty poll cycle with no actuations takes an average of 2.56 seconds, while a cycle with actuations can add 2.31 seconds or more, depending on how many devices, and are actuated and if an actuation causes more actuations to occur due to application code. During a cycle, power is consumed for receiving events, running applications to determine what actuations are needed, and performing actuations. We use the power consumption of a poll cycle as our base power consumption, as it represents the power consumption of the normal behavior of the system. Figure 12 also has the highest standard deviation of the four figures we present, because communication disruption alters with some probability how many times the functions need to attempt to communicate with the device. rollback has the highest standard deviation and cost across most graphs, because it must operate on various numbers of devices based on the chosen checkpoint. Overall, even rollback costs a tiny fraction of the power consumed during normal system operations.

Figure 13 shows the average power consumption of a fault-handling function when a power failure is injected into a heater. The amount of power consumption is also low. For the first three functions, the cost is less than the numbers for the communication-disruption scenario, as the functions cannot succeed and therefore do not consume extra time and power. The only exception is rollback, which can succeed so long as the heater does not need to be turned on for a given checkpoint. All functions still remain relatively small on power consumption, compared with this scenario’s average poll cycle costing 3.80 watts.

Figure 14 presents the average power consumption of a fault-handling function when a hardware failure is injected into a light. Among the presented scenarios, rollback has a lower cost compared to other fault-handling functions. This is because in this scenario the light must change state for a successful rollback; so the rollback simply fails to execute once this is determined. Aside from that, this graph is similar to the others with all functions consuming a small amount of power compared to the poll cycle cost of 3.68 watts.

Figure 15 presents the average power consumption of a fault-handling function when a software fault is injected into a temperature sensor. The only noteworthy difference from previous graphs is the high cost of Retry. This is a side-effect of polling the Temperature Sensor being much more expensive than polling other much simpler devices. Since Retry uses polling when attempting to resolve the fault, it also increases the cost of Retry. The higher cost of Retry would hold true for all complex devices, such as Presence, Leak, and Contact sensors that report analog rather than digital values. Despite this increase, it is still less expensive than rollback and relatively small compared to the poll cycle cost of 3.82 watts.

Another consideration in power consumption is the power cost to individual devices, which may rely on a battery or have their lifespan shortened by high power usage. To evaluate our device overhead, we collect the baseline consumption of the Arduino board with no devices powered and compare that to both when the device is powered but not doing anything and when our functions run on the device. As shown in Table 8, there is a very low cost each function incurs beyond the idle operations of the Arduino Board and devices. As can be seen in the table, the functions all have similar costs to the simple Poll and Actuate functions, and when comparing the several Watt cost of each poll cycle this cost is insignificant. It is worth noting that rollback has been left out of this table as its cost to Arduino and devices is identical to actuate, which only has a cost for light and alarm devices. Also, it is not possible to separate what is consumed by the device and what is consumed by the Arduino Board for computation using our method of power evaluation, but since both of these costs are incurred by the function call directly, their sum can be considered the direct cost incurred by a function call.

Comparing energy cost to other fault-handling solutions has the same limitations as latency comparisons. Nontheless, we compare IoTRepair’s energy consumption to a fault-tolerance solution that aims for energy efficiency [43]. This article aims to use the min-cut algorithm to minimize the overhead of saving state information to non-volatile-memory while still saving enough for good fault tolerance. They implement their system on a set of controller benchmarks and observe an energy overhead ranging from 19.9% to 65.8% with an average of 41.5%, and observe a 14.9% average overhead on Mediabench benchmarks. With our functions costing only less than 1,000 milliwatts each time they are called, the total overhead of IoTRepair across one poll cycle is a fraction of a percent. As time passes with no faults in the system, the overhead during the short periods when there are faults will shrink even more, so IoTRepair clearly beats out this method in energy efficiency.

Overall, it is clear that there is a negligible energy cost to IoTRepair. This is intuitive as compared to the lengthy time IoT devices are meant to be deployed for; a single restart or actuation is a negligible and infrequent cost.