Threat-modeling-guided Trust-based Task Offloading for Resource-constrained Internet of Things

Trust has also been used to solve a variety of problems in IoT applications. Primarily, the security protocols deployed typically need to provide identity trust, where one node can verify the authenticity of a message sent from another node. Trust has been used in a variety of areas, such as routing of messages in wireless sensor networks [19], attack detection (such as intrusion into the network) [8], and localization [32]. In these areas, trust is evaluated based on observations made about the device’s behaviors.

Trust and Reputation Systems (TRSs) are important for systems where nodes do not always behave correctly. Nodes may exhibit selective behavior, where services are correctly performed only some of the time [75]. For example, to save energy, a node may choose not to forward all messages that are routed through it. In networks where trust is derived from reputation information, the impact of manipulations of this information needs to be mitigated. An example is when nodes bad-mouth other nodes by lying and indicating they have a low trust value for another node [75].

There has been much work on developing trust models to solve these problems beyond resource-constrained systems [66, 81]. In vehicle and cellular networks the task offloading problem (which we focus on in this article) is referred to as Multi-access Edge Computing (MEC) (previously Mobile Edge Computing) [47]. A variety of solutions have been proposed for trust-based offloading in MEC systems, typically involving machine learning approaches [53, 80], linear programming (LP) [20], or game theoretic approaches [63]. However, these solutions are typically unsuitable for use in resource-constrained systems. Many machine learning models, though not all, require a large amount of memory or are expensive to compute, and techniques such as LP or game theoretic approaches would require large amounts of data to be sent to a central location to be processed into a schedule, which is costly in terms of energy. This typically means that lightweight approaches are needed that are evaluated on resource-constrained IoT devices.

The seminal example of a lightweight trust model is the BRS [42]. The BRS maintains two counters, which are parameters to the Beta distribution, the number of good events observed (\(\alpha\)), and the number of bad events observed (\(\beta\)). A trustor can calculate a trust value about a trustee via the expected value of this distribution (the ratio of good events to the total number of events). These values can be updated with more observations, allowing the belief in the trustee to be refined over time.

The BRS and other models such as those that use HMMs [27] can allow the representation of trust in a small amount of space in RAM. However, for these trust models to be effective in selecting a target for task offloading, they need a suitable system to feed them with observations and then deliver tasks to the chosen node.

There are very few task offloading middlewares that have been developed with a focus on resource-constrained devices that this work investigates. MEC approaches more commonly target cellular and WiFi networks with low-resource devices typically being smartphones or connected vehicles; it is also uncommon for work to present an actual implementation and experiments performed on real-world devices.

Aura [37] facilitates offloading tasks from resource-rich mobile clients to resource-constrained IoT devices, which is the opposite direction to which this article considers offloading. Tasks also need to be able to be divided into small pieces as they will be executed via MapReduce on the IoT devices. Such an approach may raise the lifetime of a battery-powered resource-rich device (such as a smartphone), but it will also lead to a large decrease in the lifetime of battery-powered resource-constrained IoT devices tasks are offloaded to.

An MEC approach focusing on IIoT was presented in [38]. However, this system focuses on the resources provided by the edge node and does not consider how IoT devices will be suitably implemented to offload tasks within their resource constraints.

A task offloading solution for TinyOS was briefly presented in [67]. However, it lacked details on how another device would be selected for offloading and capabilities to evaluate the success of the offload. In [58] a system was developed for sharing task compute among a group of resource-constrained devices with no access to a resource-rich device to process the task(s). These approaches have not focused on the implementation of a task offloading system and the security threats it may face. Instead, there has been a focus on how to select which device a task should be offloaded to. In this article we present a middleware that is agnostic to the mechanism being used to select an offload target; however, it focuses on providing supporting information to task offloading approaches that utilize behavioral trust.

Much work has been performed investigating attacks against TRSs [36, 75, 77], including attacks such as (1) Bad-mouthing, (2) Good-mouthing, (3) Collusion, (4) Selective Behavior, and (5) Self-promotion. However, the system that uses a TRS to perform task offloading is also important to secure. Threat modeling approaches allow the exploration of attacks against a system, its components and their interactions, plus adversaries and their goals. Attack Trees [79] are commonly used to represent an attack scenario as a tree with the root labeled with the goal of the attacker and other nodes labeled as sub-goals or attacks.

Threats are typically classified into one of two frameworks, either the CIA security properties ((1) Confidentiality, (2) Integrity, and (3) Availability) or the STRIDE [65] security threats ((1) Spoofing, (2) Tampering, (3) Repudiation, (4) Information disclosure, (5) Denial of service, and (6) Escalation of privilege). There are many threat modeling methodologies [5, 10] where the usual process is to (1) identify the system and its information assets, (2) identify the adversaries, (3) identify how the adversaries will attack the system and which vulnerabilities they may exploit, (4) calculate a measure of risk (e.g., risk = impact \(\times\) likelihood), and, finally, (5) identify mitigations to reduce risk.

Commonly used methodologies include (1) the Cyber Kill Chain [50], which focuses on threats and where controls block an attack in seven steps from reconnaissance to actioning objectives; (2) the Process for Attack Simulation and Threat Analysis (PASTA) [Chapter 7][73], which is a risk-centric analysis where risk is used to prioritize which threats mitigations should be developed for; (3) Trike [59], which takes a risk management perspective of a limited classification of threats (denial of service, escalation of privilege, and social responsibility); and (4) OCTAVE [1], which is a risk-based analysis of (a) operational risk, (b) security practices, and (c) the technology involved. The analysis focuses on assets (e.g., information and systems) key to an organization.

There have been a variety of threat analyses of task offloading systems and the specific components that the system this article proposes builds upon. For example, a survey of MEC was performed in [57] that included offloading security considerations. However, as previously mentioned, MEC solutions are not directly applicable to resource-constrained IoT devices due to those resource constraints limiting the approaches to offloading and platforms that can be used.

There has been much work on investigating attacks and countermeasures for Wireless Sensor Network (WSN) and IoT systems [3, 9, 18, 71]. This involves attacks at different layers including the physical, communication, and application layers. Attacks at these layers will have requirements, different impacts, and varying ease in which the attacks can be performed. For example, physical attacks may be difficult to perform (e.g., requiring physical access to the relevant devices) but can lead to large impacts such as via key stealing attacks or adversaries loading a custom firmware.

The majority of these analyses focus on network-level attacks, for example, attacks to routing protocols for packets in the network such as RPL [72]. Many attacks applicable to other wireless systems (e.g., eavesdropping, jamming, replay, and others) also apply to these resource-constrained systems; however, their unique aspects mean that additional attacks can have a large impact. For example, denial of sleep attacks [56] can prevent communication hardware sleeping, leading to exhaustion of energy stores and denial of service. Communication security layers such a DTLS [30] and OSCORE [Section D.1][61] have also been investigated due to systems depending on the protections these layers provide.

This system is primarily focused on providing evidence of actions taken in a behavioral trust model. The trust model is then used to select an appropriate edge node to offload a task to via an evidence-based evaluation of the edge’s past behavior. However, in order to provide a foundation for assessment of behavioral-based trust, it is necessary that nodes in the network have trust in the identities of other devices in the network.

Each IoT device is pre-deployed with a root certificate, their own certificate, and their secret key. This implementation uses the NIST P-256 elliptic curve (EC) (also known as secp256r1) for ECC keys because it provides a good level of security and the keys and signatures both take up a small amount of space (\(64 \,{\rm B}\)) compared to keys required for RSA at comparable bits of security [44]. However, a downside is that ECC operations are time consuming to compute; therefore, we aim to minimize the use of ECC operations where appropriate. Due to the large size of X.509 certificates, we use a CBOR-encoded certificate similar to XIOT [39], whose contents are shown in Figure 2. While the certificates support including the time at which they are valid, not all systems may be capable of checking the validity. This is because there may be no time synchronization protocol in use that allows IoT devices to align their local clock with a global clock. So in this system, certificates can be purged from the root node once IoT devices are expected to have run out of battery.

To minimize the number of ECC operations that IoT devices perform, a shared secret is provided to an OSCORE context, so for the majority of operations AES-CCM is used to encrypt and provide authentication of messages. In order for a node \(n_1\) to create an OSCORE context with another node \(n_2\), elliptic curve Diffie-Hellman (ECDH) is first used to generate a shared secret using \(n_2\)’s public key and \(n_1\)’s secret key. Therefore, ECC operations are only required when (1) deriving the OSCORE context, (2) digitally signing/verifying specific messages, and (3) verifying certificates.

At network deployment not all edge nodes may be known; this means there is a need for IoT devices to be able to request keys for unknown nodes from a key server on the trusted root node. The protocol for this is shown in Figure 2(b).

IoT devices need to discover edge nodes and their capabilities (i.e., what applications they are running). Discovery of these capabilities aligns with a publish-subscribe model where IoT devices subscribe to announcements of edge nodes publishing their capabilities. MQTT [7] is a pub-sub protocol designed for IoT devices; however, it has a number of downsides when integrating with this system. Primarily, MQTT uses TCP to provide reliability, which means that there would be an additional RAM cost by including a TCP library. The use of MQTT would also mean that the security mechanisms protecting CoAP messages could not be applied to MQTT messages. To mitigate this overhead on the IoT devices, we instead implement MQTT-over-CoAP, where an application on the root node translates CoAP messages into MQTT messages and vice versa. The MQTT-over-CoAP translator application communicates with a Mosquitto [46] server that provides the MQTT functionality.

There are four phases to resource-rich capability discovery, which are shown in Figure 3. The first requires IoT devices to subscribe to four topics: (1) edge/+/announce, (2) edge/+ /unannounce, (3) edge/+/capability/+/add, and (4) edge/+/capability/+/remove. The first wildcard entry (represented by +) is the edge node’s EUI-64 in hexadecimal, and the second wildcard entry is the name of the capability.

The announce topic is used for edge nodes to announce themselves to others in the network. Their lightweight certificate is included in the message, so receivers do not need to request it. The receivers will validate the certificate upon reception. The capability add topic is used for edge nodes to inform subscribers that a specific capability is being provided by that node.

The unannounce and capability remove topics are used by well-behaved edge nodes to inform subscribers that the node or a capability is unavailable, respectively. Malicious nodes may not publish these messages, so IoT devices will need to be able to handle this scenario when encountered.

An issue in trust-based selection is that when the system is starting or a new entrant joins, there is little opportunity for historical data to have been gathered and used to build a trust model. Therefore, in order to facilitate better initial decisions, stereotypes can be provided as a starting point to bootstrap trust models [66]. When an edge node announces itself, the certificate it sends contains a set of tags that provide an abstract description of the node. Once these tags are received, the stereotype for this set of tags is requested from the root node as shown in Figure 4. When choosing which IoT device to submit a task to, the stereotype with the closest set of matching tags may be used in the process of calculating the trust value for that edge node.

Trust models may incorporate a measure of reputation into their evaluation of the trustworthiness of an edge node. The reputation of a trustee is the beliefs held by other trustors in the system and it is stored in the same format as the trust model held by other trustors. When a trust model incorporates reputation, each of the IoT devices need a mechanism to disseminate their beliefs. It is important to provide non-repudiation for messages containing reputation of trustees so nodes cannot claim they had a different trust value in the past. There is also no need for confidentiality, meaning the messages can be signed and sent unencrypted.

There are multiple options for implementing dissemination of reputation information, such as (1) performing a network-wide multicast, (2) targeting specific nodes, (3) performing a one-hop broadcast, or (4) allowing nodes to request reputation information. In this implementation we focus on (3) and (4), where IoT devices perform a periodic dissemination of trust values and also allow other nodes to request reputation information from arbitrary nodes. Both of these actions are shown in Figure 5.

The messages that applications on IoT devices send to edge nodes (the task request and, vice versa, the task response) will be protected by OSCORE via encryption and authentication of the message. While some applications may not require confidentiality, a generic layer needs to encrypt the messages in order to facilitate applications that do require it. An example of the application protocol is shown in Figure 6 for an application that periodically sends a task every T seconds. When a capability add for this application is received, the application is started if it is not running.

For the first periodic action, an edge node may be selected that lacks the IoT device’s certificate. The edge node will not acknowledge the message and instead request the certificate, either when the IoT device retries whether the certificate has been retrieved or when the IoT device will eventually time out. Subsequent actions will not need to repeat this as the edge nodes will cache the certificate. In the case of failure, applications may choose to resubmit a task to an alternate edge node. This is left up to the application as it will require buffering the task to facilitate retrying it.

In this section we describe three threat actors that the system may be subject to: (1) a malicious resource-rich edge, (2) a malicious resource-constrained IoT device, and (3) an external threat actor. The edge and IoT devices may be malicious (e.g., when owned by competing organizations) or become malicious (e.g., via compromise). We describe the adversaries along the following six dimensions. Only the Tactics of TTPs are included in the description of adversary goals, as Techniques and Procedures will depend on the attacks being performed.

This section will discuss threats to interactions, data stores, and processes in general, and the full details of the threat modeling are shown in Appendix A. It is important to understand the details of a threat as to simply identify that an interaction is vulnerable to a denial-of-service attack is not sufficient to analyze mitigations to that threat.

As much of this system involves wireless communication, it means that typical threats to this kind of communication exist to interactions between different entities in the system. A spoofing threat, where an adversary impersonates another entity in a crafted packet, may have an intended impact to interact maliciously with the system as if the adversary was a genuine entity. In a tampering threat, an adversary performs a man-in-the-middle (MITM) attack or replays a previous packet. This could involve modifying information such as capability dissemination or manipulating reputation via old recorded messages. For some communications, there is a threat of repudiation, where an entity may claim they sent different information than they actually did in the past. This threat is typically associated with reputation dissemination, as a resource-constrained device should not be able to lie about reputation information it had previously sent. Due to the wireless communication, an adversary with the necessary equipment and presence can eavesdrop on communications to learn sensitive information. Also due to the wireless communication, it is relatively easy for an adversary to cause a DoS by jamming the medium to prevent communication.

There are additional threats that arise due to the resource constraints of the IoT devices. One key example that is common to data stores on resource-constrained devices is the threat posed by buffer exhaustion. As these buffers are relatively small (due to the low amount of memory available) for a large network, it will not be possible to store information on all resource-rich and resource-constrained devices. So, there arises multiple threats: an adversary will attempt to fill a buffer with low-quality information, meaning a resource-constrained device is unable to perform an offload or is unable to make a good choice about whom to offload to. Alternatively, if an eviction strategy is in use, then an adversary can take advantage of it to attempt to replace high-quality information with low-quality information.

Finally, DoS threats to resource-rich systems via a large number of requests have additional impacts on resource-constrained devices due to the cost to process these requests (which will be investigated in Section 7.2). A high computation cost to perform cryptographic operations means that fewer messages are needed to obtain a DoS and that an impact on one aspect of the system can potentially impact others (which will be investigated in Section 7.5).

There exist a variety of additional general threats to this system that are not specific to one interaction, data store, or process. One of the key general threats is that weaknesses are discovered in the implementation of the system or in the cryptographic algorithms used after deployment. These potential vulnerabilities could lead to a wide variety of attacks being performed. In resource-rich systems it is expected that software updates will be deployed to resolve these issues; however, in resource-constrained systems an OTA update mechanism is not always included. This is due to the Flash and RAM cost of supporting OTA updates and also the communication cost of performing the update. Draft standards such as SUIT [48] present a serialization format for software updates in resource-constrained devices, with a prototype implementation available for RIOT.³ Providing OTA updates presents a trade-off in terms of both the increased attack surface and the overhead to support updating, which may reduce resources available to applications or trust models.

As OSCORE is used to protect CoAP messages, it means that limitations of OSCORE apply to this system. The main threats are that CoAP headers are not confidential, as OSCORE only protects the integrity of a subset of headers and does not encrypt them. This has the potential to reveal context information to an adversary about operations being performed in the system.

We have proposed pre-deploying a public/private key pair to IoT devices lasting their lifetime. This simplifies key management and reduces the cost of managing and exchanging keys. A downside is that using ECDH to derive a shared secret once (i.e., for the lifetime of the devices) does not provide forward secrecy and no key revocation has been implemented. If required, then future standards (such as EDHOC [62]) that facilitate ECDHE will be necessary to set up OSCORE contexts.

This section describes the mitigations and notes about specific threats. Depending on the required characteristics of alternative applications, assumptions that have been made may not provide suitable protection. For example, it has been assumed that reputation should be public; however, some deployments may wish to keep reputation private. Alternatively, these notes may describe mitigations that are outside of the scope of the middleware that performs task offloading, such as jamming mitigations in low-powered wireless networks.

We now examine attack trees for a malicious resource-constrained IoT device’s goals G6 and G7 in Figure 8 and a malicious resource-rich edge’s goal G3 in Figure 9. To save space we do not include mitigations, as these are included in Appendix A. Each node in the attack tree is either a goal, a threat that a component or interaction is subject to, or a sub-goal (marked with “SubG”) to group threats. These attack trees are not intended to be exhaustive, but to include representative threats.

Figure 8 shows an attack tree with the root goal of a resource-constrained IoT node being to alter the trust values of resource-rich edge nodes. In this attack tree there are three main groups of attacks. The first involves modifying or preventing reputation dissemination for trust models that incorporate reputation. The second involves modifying the capabilities that edge nodes report. The rationale is that well-behaved resource-constrained devices will attempt to use edge nodes for unsupported capabilities and then record bad interactions that decrease trust. The third is to manipulate the stereotypes for that edge node. Finally, tasks can be prevented from being submitted, executed, or having results delivered back to the relevant resource-constrained IoT devices.

Many of these threat are mitigated by the use of either OSCORE or Group OSCORE due to the provision of integrity, authenticity, replay protection, and either confidentially (OSCORE) or non-reputation (Group OSCORE). Various mitigations exist for threats involving communication jamming (e.g., channel hopping) and DoS via the quantity of tasks submitted (e.g., filtering of tasks believed to be an attack); however, both mitigations have their limitations and will not protect against a persistent attack from a high-resource adversary.

Figure 9 shows an attack tree with the root goal of a resource-rich edge node preventing resource-constrained nodes from submitting or obtaining task results. This falls under four main categories: (1) denying service to any external resources an edge may depend upon, (2) jamming or intercepting task submission and/or responses, (3) preventing resource-constrained IoT devices from storing records of well-behaved edge nodes, and (4) manipulating the capabilities that well-behaved edges report. Again, many of these threats are protected against via OSCORE and Group OSCORE. However, threats that involve buffer exhaustion need additional mitigations, such as eviction strategies that take into account ways in which they can be attacked [13, 40].

To illustrate the operation of this system we implement an example trust model using the BRS. The trust model is solely intended to obtain a set of edge nodes that are believed to be suitable for offloading a task to, where the task will be successfully executed, and a result will be successfully returned to the IoT device within a deadline. The trust value for each metric m, edge node r, and application a is beta-distributed \(\mathcal {T}_{m}\!\left(r, a\right) \sim \text{Beta}(\alpha , \beta)\) (application specific) or \(\mathcal {T}_{m}\!\left(r\right) \sim \text{Beta}(\alpha , \beta)\) (application agnostic), where \(\alpha\) is the number of successful interactions and \(\beta\) is the number of unsuccessful interactions. The expected value of the distribution summarizes the number of successful events that have occurred as shown in Equation (2). For any application-agnostic metric, m: \(\mathcal {T}_{m}\!\left(r, a\right) = \mathcal {T}_{m}\!\left(r\right)\).

Each IoT device c maintains three sets of Beta distributions that summarize interactions with edge node r and application a:

Correctness is an application-specific and best-effort attempt to validate if a result for a task conforms to expected aspects of the result. As c will not execute the task and compare results, it must be expected that the correctness evaluation will contain false positives when evaluating malicious responses.

These three Beta distributions are updated when relevant events occur. These events need to be hooked into by the trust model. The three main events are (1) task submission, (2) task response, and (3) response quality. Other events can be defined and hooked into as needed by alternate trust models, such as in challenge-response-based trust models [15].

The overall trust value of an edge node is summarized by a weighted mean over the expected values of these distributions as shown in Equation (3), where (1) \(M(a)\) is the set of metrics that relate to application a, and (2) \(0 \le \varphi _{a,m} \le 1\) is the weight that application a gives metric m. Applications use the weight to specify the relative importance of metrics, with \(1 = \sum _{m \in M(a)}\varphi _{a,m}\).

If a stereotype \(\mathcal {S}_{m}\!\left(r\right)\) is available for edge node r and metric m, then the trust model for that metric is adjusted before calculating the summarized trust. As these trust models are initialized as \(\text{Beta}(1, 1)\), 1 is subtracted from \(\alpha\) and \(\beta\).

The individual distributions are updated as per Algorithm 1, where \(f_{a,m}^{\mathrm{opp}}\) is an application a and metric-specific m function that evaluates the opinion IoT device c has about a situation and interaction. The situation details which task was submitted, and the interaction contains information about the last interaction with the edge node. For example, a situation may be “Request route from a to b” and the interaction may be “r timed out returning a response.”

This trust model utilizes reputation information disseminated to demonstrate the cost that it incurs. As we are not investigating threats against trust models (such as peers providing false reputation information), we perform a simple combination of the peer-provided reputation information. Due to the potential for reputation to contain partial information, care must be taken in summarizing it. Equation (5) produces the reputation provided by n for resource-rich node n and application a. This is summarized in Equations (6) and (7), where peers that provide reputation on no relevant metrics are excluded. Finally, the impact peer-provided reputation has on the overall result is weighted by a factor \(\psi\) that is in the range \([0, 1]\) in Equation (8).

To choose which edge node to submit a task to, that edge node must support the application that originated the task and also have a sufficiently high trust value. A subset of candidate edge nodes that support the application \({V_{R}^{a}}^{\prime } \subseteq V_{R}^{a}\) may be known. This could be due to a lack of space in memory or IoT nodes not processing or being unable to process the relevant announcement from an edge node. For this example model we implemented a banded approach, where a sufficiently high trust value is any trust value within some distance from the maximum trust value (set to 0.25 in this work). The chosen edge node is selected randomly from the edge nodes that meet these criteria as shown in Algorithm 2.

To illustrate the operation of this system we implement two example applications: (1) environment monitoring and (2) vehicle routing. The environment monitoring application generated a task every \(1 \,\mathrm{min}\), which involves sending sensor data to an edge node. The vehicle routing application generated a task every \(2 \,\mathrm{min}\,{\rm to}\, 3 \,\mathrm{min}\) containing source and destination coordinates and expected to receive a response containing the route between those points within \(2 \,\mathrm{min}\). The routing application performs a correctness check of the result by checking that the source and destination are the first and last items respectively in the provided path. The trust model weights for these two applications are \([ 1 \quad 0 \quad 0 ]\) for Environment Monitoring and \([ \frac{1}{3} \quad \frac{1}{3} \quad \frac{1}{3} ]\) for Routing, where the vector contains \([ \varphi _{a,sub} \quad \varphi _{a,res} \quad \varphi _{a,cor} ]\). \(\psi\) for both applications was set to 0.25.

The RAM and flash usage of the implementation (shown in 4) was generated using nm on the compiled binary. This only shows the cost of defined symbols such as static variables and functions; it does not include strings. Symbols have been classified into categories to identify where the RAM and flash costs are incurred. The implementation is limited by the RAM of the IoT hardware because dynamic memory allocation is typically avoided with embedded systems, as long-term use can lead to memory fragmentation, which prevents future allocation requests from succeeding. Therefore, fixed-size buffers are chosen at compile time. In our implementation 65% of the RAM utilization comes from the buffers required to implement network access (contiki-ng/net), certificate storage and digital signatures (system/crypto), and the trust model (system/trust).

In this section, the relevant cryptographic operation costs are profiled on the Zolertia RE-Mote to understand the trade-offs of using different message protection approaches. The hardware on which these tests were performed has a clock with 32,768 ticks per second, meaning timers have a resolution of \(30.5 \,\mathrm{\upmu }\mathrm{s}\) (3 s.f.). The average costs are shown in Table 5 with 95% confidence intervals.

Results for SHA256 and ECC operations were gathered by generating a random plaintext with a random length from \(1 \,{\rm B} \,{\rm to}\, 1024 \,{\rm B}\) and then signing and verifying that plaintext. As SHA256 is performed as part of the sign operation, each sign and verify operates on a constant number of bytes, so the results are not shown per byte. Results for AES-CCM encryption and decryption were gathered by generating a random plaintext with a random length from \(1 \,{\rm B} \,{\rm to}\, 1024 \,{\rm B}\), a random \(35 \,{\rm B}\) of additional authenticated data (the maximum supported by OSCORE), a random \(16 \,{\rm B}\) key, and a random \(13 \,{\rm B}\) nonce. The plaintext was encrypted and an \(8 \,{\rm B}\) authentication tag was generated, which was then decrypted and authenticated.

These results highlight the cost difference between AES-CCM operations and the ECC operations on this IoT hardware. Performing an AES-CCM operation on a \(100 \,{\rm B}\) message is three orders of magnitude faster than an ECC operation. This performance difference is why ECC operations are only used to derive a shared secret for OSCORE and to disseminate signed reputation information, whereas AES-CCM is used to protect all other messages.

We performed a deployment with three IoT devices (wsn3, wsn4, wsn5) and two edge nodes (rr2 and rr6), where both edge nodes perform the monitoring application correctly. Two different experiments were performed, one where both edge nodes performed correctly for the vehicle routing application and another where rr2 performs correctly and rr6 always performs incorrectly for the vehicle routing application. Incorrect behavior is randomly chosen from (1) not sending a response, (2) sending an invalid response claiming the response is correct, or (3) sending a response indicating a failure. The system was run for long enough for trust values to begin to converge. Results are shown in Figure 10 when both edges are good and Figure 11 when rr2 is good and rr6 is bad. The graphs show (1) the IoT devices evaluate their trust that the edge node will execute the task (lines) and (2) the number of tasks an IoT device submits to an edge node over a time period (bars).

Figure 10(a) and 11(a) show results for the monitoring application, where trust values start high (due to stereotypes) and remain high. There are instances where trust values decrease, which may be due to transient failures such as edge nodes failing to acknowledge a task submission. The tasks submitted by the three IoT devices are distributed across the two edge nodes as no edge node has a sufficiently low trust value for them to be excluded from task submission.

Figure 10(b) and 11(b) show results for the vehicle routing application. The trust values begin at a high value due to stereotype information; however, in Figure 11(b) the trust in the two edge nodes quickly diverges due to the differing behavior. While rr6 has a trust value that is still within the maximum distance from the highest trust value, it can be chosen to execute tasks (as described in Section 6.1). However, after the trust value becomes sufficiently low, rr6 is excluded from being selected and the IoT devices only send tasks to the well-behaving rr2.

Results showing the number of bytes transmitted and received are shown in Figures 12 and 14 Figures 13 and 15, respectively. Messages have been grouped into \(6 \,\mathrm{min}\) windows. The results for IoT devices wsn4 and wsn5 are omitted as they show a similar pattern to wsn3.

The messages have been categorized where possible. Due to issues with analysis tools, not all OSCORE contexts can be decrypted, so valid messages will appear as “oscore-nd.” Not all 6LoWPAN fragments could be reassembled, so are shown as “6lowpan-fragment.” Packets marked as “oscore-nd” were for a variety of purposes, including the two applications and potentially other categories where messages could not be decrypted. “trust-dissem” packets are intentionally not protected with OSCORE, as they need to be signed and not encrypted. The implementation currently manually includes a digital signature, which will be the case until Group OSCORE is supported (as will be described in Section 8.4).

Comparing the two edge nodes rr2 (always good) and rr6 (always bad) shows why an edge node may choose to perform maliciously, as there is a greatly decreased cost in delivering correct application functionality. Edge node 2 has a higher number of messages sent and received than rr6 because by performing correctly, it needs to deliver the result of the application. For the vehicle routing application task, this means that a result of \(9 \,{\rm KiB} \,{\rm to}\, 10 \,{\rm KiB}\) needs to be delivered back to the IoT device, which involves sending 53 CoAP messages and receiving the same number of acknowledgments in the best case.

The vehicle routing application has the largest proportion of bytes sent (\(\gt\)74%) and received (\(\gt\)87%) for the three IoT devices. Trust dissemination and subscribing to capabilities are also expensive, costing 6–8% of bytes sent and 4–5% of bytes received. Packets that our analysis tools could not process (marked as “oscore-nd”) took up 7–8% of bytes transmitted and 2–3% of bytes received. This indicates a worst-case 26% overhead in transmitted bytes and 13% overhead in received bytes to facilitate trust-based task offloading. In reality for the sample applications these overheads will be lower, as some application packets were categorized as “oscore-nd.” The overhead will differ depending on the frequency of reputation and capability dissemination, frequency of tasks, and the payload sizes of tasks and their responses. Applications that require less data to be sent and received will lead to their being a greater overhead to disseminate information required for trust-based task offloading. Deployments would need to adjust the rate at which reputation and capability information is disseminated based on application needs.

In this section we investigate the impact of an attacker attempting to cause a denial of service of signature verification in order to prevent resource-constrained devices from being able to process reputation information disseminated from other resource-constrained devices. The adversary on startup signs a packet containing reputation information in a valid format and broadcasts it to its neighbors every \(300 \,\mathrm{m}\mathrm{s}\). The packet is correctly generated each time with correct CoAP headers.

In this experiment we investigate the presence of an adversary instead of wsn5. rr2 and rr6 act as well-behaved resource-rich nodes, and wsn3 and wsn4 are well-behaved resource-constrained nodes. We include results from where there are two good resource-rich nodes (2G) with an experiment duration of 2 hours and 5 minutes and one good and one bad resource-rich node (GB) with an experiment duration of 2 hours and 7 minutes for comparison.

The results in Table 6(a) show that the attack has no impact on devices being able to send reputation information. This is because the attack focuses on the buffers used to verify received packets, even though both share hardware used to accelerate cryptographic operations.⁴ The impact on receiving reputation information is shown in Table 6(b), where two configurations are investigated under attack. A2/3 is an attack on the system where both resource-rich nodes are good with a duration of 53 minutes, the size of the reputation receive buffer is 2, and the verify buffer is 3 (as shown in Table 4(a)). Here, it can be seen that the two resource-constrained nodes receive a large number of reputation messages to verify from the attacker. However, not all of the messages are successfully verified (SUCCESS), some fail as the relevant certificate is missing (NO_KEY), and others fail because there is no space in the reputation receive buffer (OOM). A high proportion of messages from the attacker are verified (33%) compared to reputation messages from other resource-constrained devices (0–6%), indicating that a DoS attack against this buffer is feasible by an obvious attack.

However, there is an additional impact that can be achieved if the system is poorly configured. For each reputation message that is queued to be verified it consumes space in two buffers, one for receiving reputation and a second for verifying messages. By default (Table 4(a)), the size of the verify buffer is larger than the reputation receive buffer, but if they have the same length (the A2/2 configuration with a duration of 50 minutes), there is an additional impact achievable. Here, an attacker can cause a DoS on both buffers and verifying a message can fail when the verify buffer is full (VER_OOM). However, as adding certificates also depends on signature verification, it means that reputation dissemination can potentially prevent or delay this. Delays in verifying and adding certificates ranged from \(2 \,\mathrm{min}\) to \(34 \,\mathrm{min}\); however, these delays may be higher in systems with lower rates of reputation and capability dissemination. These delays prevent OSCORE contexts from being created between parties and mean that no secure communication can occur between them. This is shown in Table 6(c) where only A2/2 fails to verify certificates as the verify queue is full (VER_OOM). For the majority of pairs this leads to an increase in time for a certificate to be verified, but some nodes are prevented from establishing an OSCORE context in these experiments.

This attack highlights vulnerabilities the combination of limited memory and processing power can lead to. It also raises the need for implementations of security protocols such as Group OSCORE (that will make significant use of digital signatures) to build in appropriate mitigations. For example, Group OSCORE APIs should allow CoAP endpoints to disallow Group OSCORE messages when not expected or required. Applications that only need the guarantees provided by OSCORE should be able to indicate that Group OSCORE protected messages will be rejected. Secondly, endpoints that do accept Group OSCORE projected messages will need to include protection. Heavyweight techniques (such as those used to mitigate (D)DoS in cloud environments [11]) will likely be unsuitable, so lightweight techniques [52] will need to be used instead. However, the implementation of these techniques will need to take care that the memory trade-off is appropriate.

For our second example attack, we investigate a resource-rich node who always performs badly and periodically publishes a capability remove or an unannounce with the intent for resource-constrained devices to remove the low-trust observations from their database. This is because when these messages are received (as shown in Figure 3), a resource-constrained device may remove the information—as the resource-rich device has claimed it is no longer relevant—in order to make space for new edges with capabilities matching the resource-constrained device’s.

In these experiments rr6 always performs the vehicle routing application incorrectly and we use different resetting behaviors to examine how much trust rr6 can regain. We investigate three different scenarios: (1) rr6 publishes capability remove and the IoT devices eagerly remove the capability trust information for the vehicle routing application, (2) rr6 publishes unannounce and the IoT devices eagerly remove the trust information on rr6 and all its capabilities, and (3) rr6 publishes unannounce and the IoT devices mark all the trust information for removal if needed (e.g., to make room for a new edge). Due to the low trust value of the rr6, it would typically be excluded quickly as the default parameterization of selecting which edge to offload a task to is to select from an edge with a capability where the trust value is within 0.25 of the highest trust value, so this band has been set to be within 0.5 of the highest trust value in these experiments. Reputation dissemination is also disabled to highlight the changes to the direct trust values.

The graphs in Figures 16 and 17 respectively show the results for the monitoring and vehicle routing applications in these experiments. It can be seen that in Figure 17(a), when capability remove is used to reset trust, the trust in rr6’s routing capability decreases over time with periodic spikes of higher trust. The spike is due to the data stored by \(\mathcal {T}_{\text{corr}}\!\left(rr6, routing\right)\) being reset; however, as \(\mathcal {T}_{\text{sub}}\!\left(rr6\right)\) and \(\mathcal {T}_{\text{res}}\!\left(rr6\right)\) are not reset, any incorrect results that are stored here persist and lead to the overall continuing decrease in trust.

When unannounce is used to reset trust in Figure 17(b), this gradual decrease is not observed. Instead, there is a periodic restart to the initial trust value, which decreases before the next reset. This is caused by all of \(\mathcal {T}_{\text{corr}}\!\left(rr6, routing\right)\), \(\mathcal {T}_{\text{sub}}\!\left(rr6\right),\) and \(\mathcal {T}_{\text{res}}\!\left(rr6\right)\) being deleted and reset to their initial values. One additional impact that an unannounce has is that it resets trust values for applications that are behaving well, i.e., the monitoring application. This leads to the trust in the monitoring application decreasing while the trust in the vehicle routing application rises. Potentially to discourage this attack, the stereotypes for the monitoring application should start lower so the attack causes a greater decrease in the trust in monitoring when an unannounce is published.

In Figure 16(c) and 17(c) a lazy approach is taken to the removal of edges and capabilities after an unannounce. Instead of immediately removing the information, it is marked as inactive. When the space is needed for new edges or capabilities and there is no available memory, it will be freed. These results show that publishing an unannounce has no impact, as the trust history is not removed in this scenario as no new resource-rich edges are discovered. A downside is that lazy removal complicates logic throughout the middleware, as extra checks need to be performed to ensure an edge or capability is active and objects representing edges and capabilities need to be used carefully such that memory is not used after they are freed.

These results show that the architecture in which trust offloading is performed can have an impact on the values calculated by trust models. This will be important to consider for resource-constrained devices, as network deployers will need to make a decision about information management and the ways that adversaries will be able to take advantage of it.

In this system there only exists a single root node, which is problematic from a security viewpoint as this presents a single target that can prevent functioning of the system. The reason for this single root node is due to implementation limitations in Contiki-NG, where their implementation of RPL (RPL-lite) only supports a single border router. While there is an alternative RPL implementation (RPL-classic), Contiki-NG’s documentation recommends using RPL-lite.⁵

There are two approaches to resolving this issue. The first would be to switch to using an RPL implementation that supported multiple border routers. In this case there would need to be consensus among the multiple root nodes when edge nodes publish their capabilities. The second approach would be to distribute the provision of the information that the root nodes provide, including stereotypes, certificates, and pub/sub of edge capabilities. There exist a number of distributed database techniques for these kinds of resource-constrained systems [23]; however, such an approach would need to consider the impact of losing central control over this information. For example, stereotypes would not be able to be updated and there would be no central authority on identities, so additional threats would need to be considered (e.g., Sybil attacks).

With multi-tenant IoT devices there are risks posed by per-application interaction. Operating systems such as Tock [45] (which is written in Rust) provide useful guarantees—such as memory isolation—at the language level. However, Tock currently lacks support for higher-level functionality needed to build complicated systems, such as support for RPL [2] or CoAP [64].

An optimization to reduce the cost of announce and capability messages would be to use the MQTT retain flag. This means when a message is published, that message is saved and delivered to nodes that subscribe in the future. However, a retained message may contain outdated information (e.g., an edge crashed or a capability becomes unavailable). Therefore, in this work we have chosen to be conservative and have edges periodically publish information.

Due to the use of recently published standards, there are some features of the libraries being depended upon that are not yet implemented. First, the implementation of OSCORE for Contiki-NG does not yet implement RFC 8613 Appendix B.1 [61], which means that when nodes reset, they will be unable to restart communication via OSCORE. Second, the Group OSCORE draft standard [70] does not yet have a fully working implementation for Contiki-NG, so signed and unencrypted trust packets cannot be protected by Group OSCORE. To work around this, the payload is signed; however, this will not protect the CoAP headers that Group OSCORE protects.

In this system, IoT devices evaluate a measure of behavioral trust that an edge node will execute a task and return a correct result within a deadline. Edge nodes will discover the opinions others have for them due to reputation dissemination. However, edge nodes lack context for why these beliefs are held. An edge node will not know which tasks and observations that IoT devices made that led to a poor trust value. Edge nodes should be able to request (or be informed) of the reasoning behind their trust values, facilitating explainable AI. However, this must be balanced with the difficulty of maintaining a large history of observations on the IoT nodes due to limited RAM.

In this work we have decomposed the system into individual components and interactions between those components to perform threat modeling. This has allowed the identification of ways in which such a system would be attacked and what mitigations have been provided. However, threat modeling will not identify every threat, and because we have taken a situation-agnostic viewpoint, it will not include those situation-dependant threats. Our threat modeling is also subject to the biases, relative experience, and relative knowledge of the people who have undertaken the threat modeling. To mitigate this we have involved people from both academic and industrial backgrounds to perform the threat modeling.

This middleware has been designed to be agnostic to the trust model in use that selects an appropriate edge node to offload a task to. Users of the system must choose an appropriate edge selection approach, which may not even be a trust model. The example trust model used in this work evaluates trust by building beta distributions about three aspects of task offloading. This was chosen in order to provide granularity to evaluate general edge behavior and specific behavior for applications. Alternate trust models that provide either a lower or higher granularity could both be effectively used as part of this middleware. Support is provided for stereotypes and reputation, but they do not need to be used. If reputation is used and there are concerns about threats such as collusion, then suitable collusion-resistant trust models [54] will need to be used. Further work may be needed to develop low-memory variants of these trust models.

In this work we focused on IoT devices being able to select which edge node a task should be offloaded to. This was intentional in order to eliminate threats that a centralized task scheduler would pose. However, the system could be configured such that a single edge is advertised to the IoT devices and that edge performs a subsequent offload based on the tasks that are submitted to it. This is not a recommend configuration of this system.

In addition, we proposed a single method via which an edge node would be selected based on a trust model in Section 6.2. Our implementation⁶ can be configured with a variety of alternate methods for choosing an edge node a task should be offloaded to based on the information provided by the trust model.

The middleware that we have developed is scalable insofar as the memory constraints of IoT devices allow. In Table 4(a) we specify the constants used, such as allocating space for 12 certificates and space for trust and reputation information for four edge nodes. These constants can be configured for different-size networks. If more IoT devices or edge nodes exist than there is capacity for in memory, then there are two options: (1) more capable IoT devices are required with more RAM or (2) data can be evicted. When evicting information, it is important to ensure that IoT devices can continue to offload tasks. Our future work is investigating appropriate eviction strategies for this information [13] and threats to eviction strategies [40].