NEPTUNE: A Comprehensive Framework for Managing Serverless Functions at the Edge

NEPTUNE assumes that an application app be developed and deployed as a set \(F_{app}\) of functions —as envisioned by the serverless paradigm—wrapped in lightweight virtualization technologies like containers. The developer writes the functions, and NEPTUNE automatically manages the underlying MEC infrastructure. This way, NEPTUNE manages relatively small components that allow for more flexibility and agility while scaling the system compared to traditional (e.g., monolith) and more recent (e.g., microservices) architectures.

For each function f, users must specify a requested response time \(RT^R_f\) (with \(RT^R_f \le RT^R_{app}\) ) and the memory required to properly run it on CPUs ( \(m^{CPU}_f\) ) and on GPUs ( \(m^{GPU}_{f}\) ), if possible. NEPTUNE then automatically deploys f, by creating one or more function instances, and manages its operation to guarantee set response time.

To manage the function instances deployed in containers (one instance per container), NEPTUNE employs a three-level control hierarchy, namely Topology, Community, and Node levels. A shared monitoring layer lets the different components communicate with each other. This enables the collection of performance metrics, such as response times and network latency, which are needed by the three control levels to operate correctly and precisely. The Node level and Community level do not require any prior knowledge of the hardware configurations of the different nodes, and they can manage heterogeneous nodes equipped with different types of CPUs/GPUs and with different virtual machines. Figure 2 shows a MEC topology managed by the three-level hierarchical control employed by NEPTUNE. The main NEPTUNE components are shown as dark grey boxes.

Given that function placement is a well-known NP-hard problem [41], the Topology level reduces the complexity for the other levels by partitioning the MEC topology into communities of closely-located nodes. Requests that reach a node in a community are handled within the community itself. If it is not possible, the communities are probably unbalanced and the Topology level must re-partition the topology accordingly. This way, communities are independent of one another.

The Topology level exploits a single controller that is based on an existing clustering algorithm named Speaker-listener Label Propagation Algorithm (SLPA), which was originally presented by Xie et al. [61]. The complexity of SLPA is \(O(t*N)\) where N is the number of MEC nodes and t is the maximum allowed number of iterations. Being the computational complexity linear with the number of nodes, this solution is practically usable even for large MEC infrastructures [5]. SLPA partitions the topology in a set of communities with a maximum size of MCS nodes and a maximum inter-node delay \(\Delta\) (i.e., \(\delta _{i,j}\le \Delta\) for all nodes i and j in the community). SLPA allows a single node to be shared among multiple communities. To prevent resource contention and to avoid dependencies among communities, NEPTUNE re-allocates shared nodes—if needed—to generate non-overlapping communities. Figure 2 shows a MEC network with N communities.

At Community level, a controller oversees the management of function instances. NEPTUNE exploits a dedicated controller for each community, and there is no direct communication among them. The controller is in charge of placing function instances on the community’s nodes. Given the Node level controllers are able to vertically scale function instances, no more than one instance per function type can be deployed on a single node. Given the combinatorial nature of the placement problem, the Community level controller exploits a formulation based on MIP. This formulation supports two types of functions: the ones that can be run on both GPUs and CPUs (e.g., the inference phase of machine learning applications) and the ones that can only run on CPUs. The optimization process first considers the availability of GPUs for functions that support them and then allocates the remaining functions to traditional CPUs. The objective of the controller is to minimize, at the same time: (i) network delay by placing function instances as close as possible to where the workload is generated, and (ii) number of MEC nodes required to serve the workload. However, there exists a tradeoff between these two factors since fewer nodes may limit the ability to place function instances close to the workload. To address this inherent tradeoff, NEPTUNE provides users with the flexibility to prioritize one of the factors according to their specific requirements.

The Community level also computes a set of routing policies to permit each node to forward a well-defined and deterministic percentage of the workload to other nodes. If a node is equipped with GPUs and runs one or more computation-intensive applications, the set of routing rules also defines the percentage of local requests that are handled by GPUs and those executed on CPUs. NEPTUNE always tries to fully exploit the GPUs and uses CPUs only when they are saturated.

By computing placement actions, the Community level can add or remove function instances on the MEC nodes (horizontal scaling). For example, if the resources of node i are not sufficient to handle the entire workload \(\lambda _{f,i}\) of function f, the Community level creates a new instance (of f) on a node j close to i to handle the requests that cannot be served on i.

NEPTUNE ensures minimal service interruption when new placement actions are computed. Existing functions are deleted only after serving the requests they were handling (within a maximum timeout), and nodes are turned off only when all function instances placed on them are either removed or migrated to another node.

Figure 2 details Community 1. Each node hosts a set of instances of the three functions introduced before (☐, ○, \(\Diamond\) ) and defines proper routing policies. For example, Node N1 runs an instance of ☐ and ○. Its routing policies state that 100% of the workload for function \(\Diamond\) must be forwarded to Node N3 since it is the only node in the community that has a running instance of \(\Diamond\) . The requests for functions ☐ and ○ must be shared, according to the given percentages, with nodes N2 and N3, since they are running other instances of the two functions. Given that Node N1 is equipped with a GPU and function ○ can be accelerated, an additional routing policy defines that within the 50% of requests for ○ that are handled locally, 95% of them are executed on the GPU while 5% on CPUs. Percentages are recomputed every time the Community level is re-executed.

Unlike the first two control levels that mainly target network delay (D), the Node level is dedicated to allocating CPU cores to function instances on the nodes in charge of processing them. In particular, it aims to optimize the sum of queue Q and execution E times, that is, the handling time or QE. Each function instance is equipped with a dedicated Proportional Integral (PI) controller that continuously provides vertical scaling of CPU cores. This means that function instances are reconfigured on the fly without the need of restarting them. This way, service disruption, which usually occurs in other solutions [2, 45], is avoided.

The PI controller is fed periodically with monitoring data on the performance of the controlled function instance. In turn, it continuously computes the optimal CPU core allocation to make the handling time meet a user-defined set point. These controllers are independent of each other, and they are not aware of the resource allocations computed by the others. This means that the sum of the resource allocations computed by the PI controllers of the different function instances that are running on a node can exceed the capacity of the node itself. To avoid resource contention a dedicated component, called Contention Manager, gathers proposed CPU allocations and, if needed, scales them down proportionally to fit available node resources. Figure 2 illustrates the details of Node N1. The CPU cores of each running function instance (☐ and ○) are vertically scaled by a dedicated PI controller, and resource contentions are handled by Contention Manager.

The three control levels work at different time frequencies but in a coordinated way to avoid potential interference. PI controllers (reasoning) and vertical scaling (actuation) are extremely fast so that the Node level can operate every few seconds and quickly react to spikes in the workload. The Community level calculates function placement and routing policies every few minutes so that the underlying infrastructure is fully exploited. Finally, the control period of the Topology level is generally slower. This level only intervenes when the two other control levels struggle in meeting foreseen response times. In case a node fails or is added to the network, the Topology level is automatically activated to properly find a new efficient network partitioning. The hierarchical architecture allows NEPTUNE to scale efficiently: it maximizes the independence among control levels and the introduced communication overhead is negligible.

NEPTUNE first allocates functions on nodes equipped with GPUs, and then considers CPUs. The formulation presented below is generalized since the two optimization processes are similar. Some of the employed data are resource-specific (e.g., \(x_{f,i,j}\) , \(m_{f}\) ): while in Table 1 we use GPU or CPU superscript to differentiate them, the rest of the section omits these superscripts for the sake of simplicity (differences between the two processes, when present, are explicitly stated).

Network delay and cost minimization. The first optimization step aims to place function instances in a community \(C \subseteq N\) and generate routing policies that minimize both network latency and the overall cost of running the MEC infrastructure.

This optimization problem uses three decision variables: \(x_{f,i,j}\) , \(c_{f,j}\) and \(y_{j}\) . The first one ( \(x_{f,i,j} \in [0:1]\) ) represents the fraction of incoming f requests⁵ ( \(\lambda _{f,i}\) ) that node i routes to node j (i.e., routing policies). The second one ( \(c_{f,j}\) ) is a boolean variable that is set to true if an f instance is placed onto node j (i.e., placement), and false otherwise, and the third one ( \(y_{j}\) ) is also a boolean variable that is set to true if any (GPU/CPU) function instance is deployed onto j.

The objective function aims to minimize the sum of two contributions: the total network delay (ND) and the cost of the MEC infrastructure (CO), that is, the sum of the costs of the running nodes. ND is defined in Formula 1 as the total sum of each inter-node delay \(\delta _{i,j}\) between node i and j multiplied by the incoming workload in i ( \(\lambda _{f,i}\) ) and the fraction of such a workload that is routed from i to j ( \(x_{f,i,j}\) ):

If we only used the first two factors ( \(x_{f,i,j}*\delta _{i,j}\) ), we would have assumed that the incoming workload was distributed equally among nodes (i.e., each node receives the same amount of requests). For this reason, we introduced the per-node incoming traffic ( \(\lambda _{f,i}\) ) as a weight. This way, the higher the workload in a node is, the more important its optimization becomes. Note that the workload \(\lambda _{f,i}\) that is to be scheduled on CPUs is recomputed by taking into consideration the requests already assigned to GPUs.

The cost of the infrastructure is computed as the sum of the cost of each active node:

The overall objective function is described by Formula 3. As the two contributions assume different value ranges, a simple sum of the two factors could lead to inappropriate values. To address this issue, we normalize both factors, and scale them between 0 and 1 based on their minimum and maximum values. Furthermore, to let users handle the tradeoff between network delay and cost, NEPTUNE provides parameter \(\alpha \in [0, 1]\) to configure the importance of each contribution (i.e., \(\alpha =1\) minimizes the overall cost of running MEC nodes, \(\alpha =0\) minimizes network delay, and \(\alpha =0.5\) assigns equal weights to both contributions).

To properly minimize network delay, we must define a set of constraints. First, the workload cannot be routed too far from where it enters the system. A parameter \(\phi _f\) defines the maximum allowed network latency of each f request:

Second, if a node is configured to execute requests for a certain function f, either directly received or forwarded by other nodes, then it must run an f instance to serve them:

Third, the sum of the memory requirements of the different functions ( \(m_f\) ) placed on j must be less than the capacity of node \(M_j\) . Furthermore, a fraction of the node’s memory, equal to the maximum memory requirements of all functions, is always reserved to ensure that new functions can be created before existing ones are removed (as detailed in Section 3.2).Fourth, the computational power of the GPUs and CPUs of a node must be sufficient to handle the incoming workload so that the sum of the average GPU (CPU) resource utilization ( \(u_{f,j}\) ) for each processed f request is lower than or equal to the capacity of the node ( \(U_j\) ):Fifth, all the requests toward a node i must be either executed locally or forwarded to a node j, which means that the generated routing policies cover all the incoming traffic for each node:when \(i = j\) , \(x_{f,i,j}\) defines the fraction of requests executed locally on i (using CPUs or GPUs), while when \(i \ne j\) , we consider remote executions.

Sixth, a node j is active if there is at least one instance (to be) deployed onto j:

The final constraint serves to restrict the total cost of the infrastructure. In particular, NEPTUNE allows users to specify a maximum budget K to ensure an efficient selection of the nodes to keep active. The constraint is expressed as

Note that, after assigning the workload to GPUs, the budget K available for CPUs is recomputed by taking into consideration the budget spent for GPUs. Similarly, when a node i is enabled when considering GPU workloads (i.e., \(y^{GPU}_{j} = 1\) ), it is also considered to be enabled when dealing with the remaining CPU workload (i.e., \(y^{CPU}_{j} \gt = y^{GPU}_{j}\) ) and its cost \(k_j\) is set to be zero (since its actual cost have been already subtracted from K while considering GPUs.

This formulation finds the best placement and routing policies that minimize the overall network latency and the cost of running the MEC infrastructure. However, the optimal solution may require several disruptive deployment actions (i.e., creations, migrations, or deletions). To address this problem, the second step of the optimization process allows for reducing service disruption and improving the result.

Disruption minimization.

The second step aims to generate function placements and routing policies that minimize service disruption. This step looks for solutions that are slightly less efficient than the optimal one computed in the first step but that minimize the number of disruptive actions to execute (e.g., function migration). To do that, the second step keeps the constraints defined in Formulae 4- 10 and adds:

This is to say that the final placement must have a placement quality in the interval \([O_{best},\) \(O_{best} * (1+\epsilon)]\) , where \(O_{best}\) is the optimal placement produced in the first step (i.e., the result produced by Formula 3), and \(\epsilon\) is an arbitrarily small value that (slightly) worsens the solution in terms of network overhead and infrastructure cost. Parameter \(\epsilon\) essentially quantifies how much we are willing to deviate from the optimal solution in terms of network delay and cost to lessen the impact on the system. It represents a tradeoff between system performance and service disruption. For instance, if \(\epsilon\) is set to 0.05, we are allowing the system to select a solution that is up to 5% worse than the optimal in terms of network delay and cost but potentially involves fewer migration actions.

We also consider the amount of created, migrated, and deleted f instances between two configurations, that is, between the current placement ( \(c^{old}_{f,i}\) ) and the one to be actuated ( \(c_{f,i}\) ). \(DL_f\) and \(CR_f\) define the number of deleted and created instances, respectively, computed as the maximum between 0 and the differences between the instances in the two configurations:

The number of migrated instances (in the new placement), that is, the amount of function instances moved from one node to another, is defined as the minimum between created \(CR_f\) and deleted \(DL_f\) :

The new objective function is then defined as

The goal of the objective function is to minimize the number of migrations ( \(MG_f\) ). Creations and deletions are needed, to avoid under- and over-provisioning. The two factors \(\tfrac{1}{DL_f + 2}\) and \(\tfrac{1}{CR_f + 2}\) , which are always lower than 1, allow for selecting the best solutions among the ones with the same number of migrations but with a different number of deletions and creations.

The combined effect of the new objective function and the constraint defined in Formula 11 guarantees the generation of placements and routing policies with close-to-optimal placement and with the minimum amount of disruptive actions. The control-theoretical planners at Node level are then in charge of vertically scaling containers as needed.

When a new function placement is computed, NEPTUNE must handle the transition from the existing placement to the new one. This transition, if not managed correctly, can lead to service unavailability (e.g., a node is shut down before the incoming requests are processed or routed elsewhere). NEPTUNE includes a transition algorithm that minimizes service interruption and cold starts. Specifically, the algorithm focuses on (i) removing a function instance only when it served all incoming requests, (ii) ensuring there is always at least a function instance that is serving requests, and (iii) shutting down nodes when no function instances are deployed onto them.

Algorithm 1 describes procedure StateTransition employed by NEPTUNE for managing state transitions. The algorithm assumes that nodes always be provided with enough resources to create at least a new function instance. This is guaranteed by Equation (6), which ensures that each node maintains sufficient free memory to accommodate an instance of the most resource-intensive function. First, the algorithm computes the nodes that must be activated or deactivated by comparing the previously activated nodes \(\lbrace y^{old}_{j}\rbrace\) with the newly computed ones \(\lbrace y_{j}\rbrace\) (line 2). At line 3, function instances that need to be managed are split into three different sets: delete set (DltSet), create set (CrtSet), and migrate set (MgrSet). The first two sets include \((targetNode, functionType)\) pairs that identify the node where a given function type must be deleted/created. The third set includes \((sourceNode, targetNode, functionType)\) triples that define the function type that must be migrated from a source node to a target node.

From an availability perspective, functions in the delete and create sets are less critical compared to those in the migration set. Creations can produce unavailability only when cold starts are not properly managed. Deletion of a function instance may affect requests that are currently processed by the system. However, as a consequence of Equation (8), there must always be at least one instance in a community for each function type. This means that there cannot be a set of deletion actions that remove all the instances for a given function type. Conversely, migrations may induce more severe temporary service unavailability when all existing instances of a function are required to move to different nodes.

Next, the algorithm synchronously activates the nodes that should host new function instances, and that were not activated in the old placement (line 4). Then, for each pair in DltSet, NEPTUNE calls asynchronously (i.e, without waiting for its termination) procedure SafeDelete (lines 5–7) that is reported at lines 21–24. In this procedure, the routing policies for a given target node and a function type are removed from all the other nodes so that the function instance ceases to accept new requests (line 22). Moreover, before terminating the function instance, NEPTUNE employs a grace period to serve remaining requests (line 23).

Similarly, in lines 8–13, migrations are handled by calling, asynchronously, procedures SafeCreate (on the function type and target node) and SafeDelete (on the function type and source node). Procedure SafeCreate is reported in lines 25–29. For each function to create, the procedure creates an instance, configures the runtime and downloads needed libraries, and finally allows for incoming traffic by setting up the routing policies. This way, cold starts are mitigated since requests are forwarded to newly created instances only when they are ready to serve requests.

Given that migrations necessitate the creation and subsequent deletion of function instances, the algorithm must wait for these migrations to complete (line 14) before initiating the creation of additional function instances (lines 15–17). This sequence ensures ordered processing and maintains nodes with enough space to host new function instances. Finally, after waiting (if needed) for the completion of delete actions (line 18), the nodes that do not host any instance are deactivated (line 19).

The Node level aims to control QE, that is, the handling time, defined as the sum of the queue time Q and execution time E; while network latency D is already optimized by the Community level. QE is subject to multiple factors, such as fluctuating workloads or changes in the execution environment. The feedback loop in our controllers allows them to handle these changes at runtime by continuously changing the CPU cores allocated to function instances. This way, Node level controllers help meet set response times given a placement produced by the Community level. If the Node level cannot keep QE under control, it means that the nodes are over saturated and the Community level is triggered to compute a new placement and routing policies.

The Node level controllers borrow from our previous work on control-theoretical solutions for dynamic resource management [6] and from other similar works [16]. They exploit lightweight PI controllers to quickly (re-)allocate CPU cores to each function. They feature a fast control period because they can compute the next state of the system in constant time, and they also provide a-priori formal guarantees.

Each function instance is controlled by a dedicated PI controller. The feedback loop monitors the average value of QE, calculates the new core allocation, and actuates it by reconfiguring the container that wraps the function instance on the fly. More formally, given a set point \(QE_{f,desired}\) (the desired handling time), the controller periodically reads the current value of \(QE_{f,j}\) (controlled variable) from the monitoring infrastructure, that is, the value of \(QE_f\) measured on node j, and computes the difference between desired and actual values. Note that, since the controllers aim to keep \(QE_{f,j}\) as close to \(QE_{f,desired}\) as possible, the set point should be set to a lower value than \(RT^R_f\) , the required response time for function f.

PI controllers are reactive. They compute the mismatch between the set point and the monitored value and compute the number of cores that the function should use to minimize the error. Algorithm 2 describes the logic of PI controllers.

Line 2 starts the computation by calculating the error, denoted as err, which is the difference between the inverse of \(QE_{f,desired}\) and \(QE_{f,j}\) . This error serves as a measure of how far the current performance is from the desired one. Line 3 retrieves the current core allocation (cpu) for the function instance.

Line 4 calculates the integral contribution from the previous control step, \(int^{old}\) as the difference between the current core allocation, cpu, the integral gain, \(g_{int}\) (a tuning parameter), and the previous error \(err^{old}\) . Line 5 computes the new integral contribution, int, by multiplying the current error, err, and the integral gain, \(g_{int}\) , and adding the previous integral contribution, \(int^{old}\) . The integral contribution captures the accumulated past errors, and allows for a correction based on the system’s history. Then, line 6 calculates the proportional contribution, prop, by using the current error, err, and the proportional gain, \(g_{prop}\) (tuning parameter). The proportional contribution offers an immediate response to the current error to rapidly bring the system performance closer to the desired state.

Next, line 7 derives the core allocation for function instance f deployed on node j ( \(cpu_{f,j}\) ). This allocation is the sum of the integral and proportional contributions, adjusted to fall within the bounds of the minimum and maximum allowed core allocations, \(cpu_j^{min}\) and \(cpu_j^{max}\) , respectively (line 8). With this new allocation, the system gears up for the next iteration, set to correct the error and maintain the desired set point. Finally, the previous error value, \(err^{old}\) , is updated with the current error at line 9 to ensure that the algorithm reflects the most recent state of the system.

The PI controllers dedicated to function instances that share the same underlying node j are independent of the others. Therefore, the computed allocations \(cpu_{f,j}\) are not immediately enacted since they could outreach the node’s capacity. Thus, the allocations of the different f instances that run on node j are processed by the Contention Manager (one per node), which calculates feasible core allocations \(cpu_{f,j}^{feas}\) . If the sum of \(cpu_{f,j}\) fits the node’s capacity, they are actuated with no changes. Otherwise, the Contention Manager scales them down according to a customizable heuristic. By default, the allocations are scaled-down proportionally, as follows:

K3S (Kubernetes) adopts a distributed architecture with a master node (K3s-server) that manages a set of workers (K3s-agents) in charge of running containers. The master node can be replicated to increase availability; all nodes use a daemon called kubelet to ensure that containers be up and running.

Kubernetes uses resources to represent both the underlying infrastructure and applications. Some resources are built-in (e.g., pods, nodes, services), but additional ones can be defined by users by using Custom Resource Definitions (CRDs) (e.g., ad-hoc monitoring, custom routing policies). All resources (built-in and custom ones) are characterized by a current state and a desired state.

Kubernetes manages resources by using the pattern controller. Users declare their goals (e.g., the desired response time) by configuring the desired state of resources, and Kubernetes employs controllers, which are components that implement control loops, to make the current state of resources move closer to their desired state. One can deploy custom controllers as well as manage CRDs created by users. For example, a pod, which is the smallest resource that can be deployed in the infrastructure and wraps one or more containers into a single isolated environment, can be configured with a minimum and maximum amount of computing resources assigned to it. Then, a built-in controller named kube-scheduler periodically checks for unscheduled pods and assigns them to nodes to let them have the amount of resources specified in the configuration.

To help controllers achieve their goals, Kubernetes provides a monitoring system, called metric-server. For instance, the kube-scheduler employs a scoring algorithm that consumes data from metric-server (e.g., CPU or available memory) to optimize the placement of pods. The monitoring system and the scheduler can be replaced by custom components to accommodate a larger variety of metrics (e.g., response time, network delay) and scheduling strategies (e.g., NEPTUNE’s function placement). Hardware heterogeneity, and thus GPUs, is managed by a dedicated built-in component called device plugin. This plugin assigns a GPU to a pod at a time, which holds the ownership for its whole lifetime.

Figure 3 shows the architecture of our prototype⁶. The prototype assumes that each node corresponds to a K3s-agent running in the cluster. Each control level is embedded in a dedicated component that implements it. In particular, component Topology controller dynamically splits the network using the SLPA algorithm, component Community controller is in charge of generating the function placement and routing policies for each community, and component Node controller embeds all the PI controllers that perform dynamic CPU provisioning at Node level.

Since existing monitoring solutions do not provide means to monitor and collect the fine-grained metrics required by NEPTUNE, we implemented a custom monitoring system that exploits the Sidecar pattern⁷. This pattern aims to create two or more containers in the same pod where one embeds the application logic, and the others perform non-functional tasks such as monitoring, logging, and configuration management without requiring modifications to the running application. Thus, the prototype employs a custom lightweight proxy (as sidecar) that is automatically injected inside all function pods, intercepts requests, and collects required metrics. Monitoring proxies push collected data into component Metrics Database, which is used by the controllers to be informed of the performance of the system.

The Topology controller periodically checks for changes in the node network (e.g., when a node is added or removed) and executes the SLPA algorithm to reorganize nodes into communities. The algorithm is configured using a CRD that allows one to specify the maximum size of a community, the maximum delay between the nodes in a community, and the maximum number of iterations of the SLPA algorithm.

The functions can be deployed using the CRD Function provided by OpenFaaS. The user must provide the function name, the container image to use, the maximum network delay, if it can exploit GPUs, and the amount of GPU memory it needs. If not specified, GPU acceleration is disabled by default.

NEPTUNE provides the CRD ServiceLevelAgrement to set the desired response time \(RT_f^R\) of the function. This resource allows one to specify the minimum and maximum amount of resources to be assigned to function instances, and also which container (of a pod) the Node level should manage when function pods are composed of multiple containers.

Once the CRDs Function and ServiceLevelAgrement are created, the Community controller addresses the new function in the placement process. Since communities are independent, at least one instance of the (new) function will be created in each community. The placement takes into account the functions that require GPUs and creates pods able to exploit the dedicated hardware. Whenever a new placement is computed, NEPTUNE routing policies are written in an additional CRD named CommunitySchedule, which is then consumed by the load balancers. In particular, the CRD CommunitySchedule is automatically managed by NEPTUNE, and no manual intervention is needed.

State-of-the-art load balancers are not designed to handle frequent changes in routing policies and they must be restarted every time the internal configuration is modified. For this reason, we developed a custom load balancer that is able to periodically change its routing policies without service interruption. The prototype deploys an instance of this load balancer on each node. When the requests for a deployed function f reaches a node j, the load balancer (running on j) collects the arrival rate \(\lambda _{f,j}\) and the average response time \(rt_{f,j}\) , and stores these data into the Metrics Database. With these data, the Community controller may (i) move the instances closer to the traffic’s source, (ii) adjust function instances according to the workload, and (iii) properly update routing policies by using the formulation described in Section 3.

The heuristic used by the kube-scheduler to deploy containers could interfere with the logic of the Community controllers. For this reason, our prototype uses a custom scheduler, implemented as a sub-component of Community controller, which directly enacts the newly computed placements. The prototype supports a seamless evolution of the system. Old instances are not deleted until the new ones are ready to serve requests. Furthermore, the deletion occurs using graceful termination periods, giving time to all instances to serve pending requests. Then, load balancers update their routing policies to guarantee service availability: whenever new routing policies are rolled out, the prototype waits until the new pods are available before replacing the old ones.

Each Node controller is in charge of scaling the CPU cores associated with each function instance (vertical scaling). The current standard distribution of Kubernetes does not allow one to change allocated resources without restarting pods, a process that sometimes can take minutes. This could compromise the Node level, which would become incapable of handling bursty workloads. As a result, the prototype augments Kubernetes with the Kubernetes Enhancement Proposal (KEP) 1287 that implements In-Place Pod Vertical Scaling⁸ and allows pods’ resources to be changed without restart. This enables Node controllers to operate at a faster pace.

To achieve coordination and scalability, our prototype minimizes communication overhead and dependencies among components and controllers. Each controller continuously monitors the desired infrastructure state described in the CRDs and reconciles it with the current state independently. When information (e.g., response time) needs to be shared across different controllers, data are written in a shared storage (i.e., Metrics Database) and, in turn, are read asynchronously by other components.

The prototype uses nvidia-docker, a container runtime that enables the usage of GPUs inside containers. However, by default, GPU access can only be reserved for one function instance at a time. This prevents the full exploitation of GPUs and limits the possible placements produced by Community controllers. To solve this problem, the prototype employs a device plugin ⁹ developed by AWS instead of the default one. This plugin abstracts a GPU by creating multiple Virtual GPUs (vGPUs), each one reserved to a function instance, and then it makes use of the Nvidia Multi-Process Service¹⁰ (MPS), a runtime infrastructure designed to transparently allow GPUs to be shared among multiple processes (e.g., containers), to enable the fractional allocation of GPUs.

When available GPUs are not enough to carry out all requested computations, we create an additional container for the same function, which only exploits CPUs (as Function 1 in Figure 3). This replication is mandatory since the runtime solutions dedicated to GPU processing (e.g., TensorFlow serving¹¹) do not allow the same container to exploit both GPUs and CPUs.

In total, the prototype comprises three custom resources and six controllers for optimal function placement, smart routing policies, request forwarding, and monitoring. The prototype requires the installation of 2 external components (i.e., AWS Plugin and Nvidia Multi-Process Service) for shared and transparent access to GPUs, and it needs to enhance K3s with KEP 1287 to enable restart-free vertical scaling.

Infrastructure. To emulate a MEC Topology, we employed a geo-distributed cluster of AWS EC2 virtual machines. We deployed the virtual machines using both different regions and availability zones to have significant network delays that resemble the ones in an edge environment. We used four different areas, each corresponding to an AWS region: Area A to eu-west (Ireland), Area B to us-east (Ohio), Area C to us-west (Oregon) and Area D to ap-northeast (Tokyo). Within the same area, the nodes were deployed onto different AWS availability zones to avoid close-to-zero network delays. The average network delays between each pair of areas are computed as the round trip times of an ICMP [40] (Internet Control Message Protocol) packet. Obtained results are shown in Figure 4. The average inter-area delays between nodes in two different regions range from 50ms and 200ms, while intra-area delays between nodes within different availability zones are between 1ms and 2ms.

To assess the performance of NEPTUNE we used a large variety of nodes. As master node, we used a c5.2xlarge instance (8 vCPUs, 16 GB memory), while as worker nodes we employed instances of different virtual machines. To represent MEC nodes with few computing resources, we deployed the workers on m5.xlarge instances (4 vCPUs, 16 GB memory) and c5.xlarge instances (4 vCPUs, 8 GB memory). Nodes with many resources were represented by c5.9xlarge instances (36 vcpus, 72 GB memory), and we employed g4dn.xlarge instances (4 vcpus, 16 GB memory, 1 GPU) as GPU-accelerated nodes.

NEPTUNE settings. The Topology controller was configured to run every 10 minutes and when the cluster was subject to a topology change. The Community controllers recompute the function placement and the routing policies every minute. Node controllers were configured with a control period of 5 seconds. One can also use faster control loops, but they may lead to inconsistent resource allocation updates since K3S resource states are stored in a remote database. The integral gain \(g_{int}\) of PI controllers was set to 25 and the proportional gain \(g_{prop}\) to 50 in almost all cases and when different values were used, we report them explicitly.

Applications. In our previous work [4], we evaluated NEPTUNE using an application (sock-shop¹²) composed of multiple functions that are dependent on one another. In this article, we conducted experiments with a new set of functions as reported in Table 2. We borrowed the functions from the literature [12, 42]. Overall, the set of employed functions is characterized by multiple memory requirements (ranging from 500MB to 4GB) and cold start times (from a few seconds to minutes).

The first benchmark is SeBS [12], a set of independent serverless functions widely employed in the literature. As shown in Table 2, we selected some of the functions of the benchmark, and we made small changes to make them compatible with the OpenFaas function interface (e.g., the addition of health check endpoints). The second application is resnet [56], a machine learning inference application. Resnet was deployed on top of Tensorflow Serving and can run using either CPUs or GPUs.

Collected metrics. We employed multiple metrics to evaluate the performance of NEPTUNE. In particular, we collected (i) response time (ms) to measure the end-to-end delay for an edge-client to send a request and receive the corresponding response, (ii) response time violation rate (% of requests) to measure the percentage of requests that were not served within the required response time \(RT^R_f\) , (iii) network time rate (% of response time) to quantify the amount of time spent by a request in the network (e.g., being forwarded from one node to another) with respect to the amount of time taken to process the request, (iv) allocated cores (millicores or thousandths of a core) to quantify the amount of resources allocated to each function.

To present consistent and statistically relevant results, each experiment was run 5 times. We use \(\mu\) to denote observed average performance and \(\sigma\) to represent standard deviation.

Workload shapes. We have designed multiple workloads to accurately simulate various user behaviors, encompassing both synthetic scenarios and real-world traces and datasets. The first synthetic workload, denoted as Ramp(MinUsers, MaxUsers, StartTime, RampTime, Duration), captures the gradual increase in user requests over time. Starting from an initial MinUsers number of users (concurrent requests), the workload remains constant for StartTime seconds. Subsequently, over a span of RampTime seconds, the workload steadily ramps up to reach the maximum number of users, MaxUsers. Following this increase, the number of users remains consistent for the remaining Duration of the experiment to allow us to thoroughly assess system performance at steady state.

The second synthetic workload we employed, named Syn(MinUsers, MaxUsers, StartTime, Period, Duration), replicates user behaviors through a sinusoidal pattern, and creates a dynamic fluctuation of user counts between MinUsers and MaxUsers. Initially, the workload begins with a period with a constant number (MinUsers) of users, followed by a sinusoidal shape that extends for the entire Duration of the experiment. Parameter Period determines the duration of a complete sinusoidal cycle, and allows us to evaluate the system’s robustness under varying user loads.

In addition to synthetic workloads, we included traces¹³ generated from three distinct real-world datasets. The first workload \(W_{cs}\) is generated from Cabspotting [39], a dataset containing GPS traces from 500 taxi drivers operating in San Francisco over one month. The second workload \(W_{td}\) is generated from T-Drive [62], a dataset featuring more than 10,000 taxi drivers in Beijing over a timeframe of one week. The third workload \(W_{tel}\) is generated from a dataset called Telecom dataset [18], sourced from Shanghai Telecom, which contains records of internet access through more than 3,000 base stations by almost 10,000 mobile phones, over a six month period.

To adapt these real-world traces to our MEC topology, we undertook three pre-processing steps. First, we normalized the geographic positions of the users, spreading them across the entire globe to align with our execution environment. This adjustment ensures that our evaluation captures a representative distribution of user locations and their associated resource demands. Second, we introduced a think time for each user to simulate realistic user behavior and workload fluctuations. The think time accounts for user interactions, delays, and idle periods, and aligns the workload characteristics to real-world scenarios. Third, selected traces were originally collected over different timeframes (from one week to six months). For this reason, we normalized the timestamps of all traces such that the workload generated from these traces was characterized by the same duration.

Competitors. We compared NEPTUNE against different well-known approaches. To evaluate the autoscaling capability, we confronted NEPTUNE against two autoscaling solutions: Horizontal Pod Autoscaler¹⁴ (HPA) and Vertical Pod Autoscaler¹⁵ (VPA). HPA provides horizontal autoscaling and employs threshold rules to actuate resource provisioning. VPA also adopts threshold rules, but it scales function instances (vertically). Note that VPA does not offer an in-place vertical scaling solution (as the one provided by NEPTUNE): it requires that function instances be restarted to change allocated resources.

Additionally, we compared NEPTUNE against three different approaches for placement and routing, namely VSVBP [26] (Variable Sized Vector Bin Packing), MCF [27] (Most Capacity First) and CR-EUA [34] (Criticality-Awareness Edge User Allocation). The first two (VSVBP and MCF) are popular approaches often used in the literature as baseline for edge resource allocation solutions [20, 33, 55], while CR-EUA is a recent extension of the previous approaches for safety-critical, low-latency applications. Specifically, VSVBP maximizes the number of allocated requests and minimizes the number of active edge nodes while ensuring the response time of the deployed services. MCF models the problem in a similar way to VSVBP, but it handles dynamic response time requirements and it proposes a faster but sub-optimal algorithm. CR-EUA proposes a solution for safety-critical applications that aims to maximize the number of requests processed with the highest level of criticality. We implemented competitors’ approaches in the Community level by replacing our formulation. To provide a fair comparison, the in-place vertical autoscaling feature, provided by NEPTUNE at the Node level, was enabled for all the competitors.

The first set of experiments aimed to evaluate how NEPTUNE allocates resources with respect to popular autoscaling approaches (i.e., HPA and VPA). These solutions can control duration QE by increasing or decreasing the amount of computing resources allocated to a function. Since this set of experiments only focuses on evaluating the effectiveness of resource allocation, we decided to use an infrastructure with a single worker node to reduce the measurement noise that network delays might introduce. The master node was deployed on top of a c5.2xlarge virtual machine in Area A, while the worker node used a c5.9xlarge virtual machine. These experiments employed the functions provided in SeBS [12]. The benchmark covers several functions: from ones used for multimedia and utilities to the ones for web apps and scientific computations. We configured the functions as reported in Table 2. We employed a sinusoidal workflow configured with \(MinUsers=1\) , \(MaxUsers=50\) , \(StartTime=60s\) , \(Period=1200s\) , and \(Duration=1200s\) .

Table 3 shows obtained results. We can observe that NEPTUNE, in most cases, was able to allocate enough resources to achieve the desired response time. The average and median response times were close to QE and well below \(RT^R\) , the desired response time. For example, if we consider functions compression and graph-bfs, they achieved an average response time equal to \(79.5ms\) and \(105.6ms\) , respectively. The response time violation rates were also really low: \(3.7\%\) for compression and less than \(0.9\%\) for graph-bfs. Similar considerations can be made for three other functions with the same response time requirements: dynamic-html, graph-mst, and thumbnailer.

The table also shows the results obtained by using HPA and VPA. Among the three approaches, VPA was the worst. Due to design limitations in Kubernetes (illustrated in Section 5.1), VPA requires that pods be restarted to change allocated resources. This negatively impacts the reactiveness of the approach and worsens the effectiveness of autoscaling capabilities. The results in the table show that VPA achieved reasonable response times only for function compression: an average response of \(51.3ms\) and a response time violation rate lower than \(0.3\%\) . As for the other functions, VPA failed to ensure the desired response time and most of the requests ended in response violations. For example, functions graph-bst, graph-mst and thumbnailer were characterized by a response time violation rate equal to \(100\%\) , \(99.8\%\) and \(99.0\%\) , respectively.

On the other hand, HPA was more effective in ensuring response times than VPA. For instance, the average response times for function compression and dynamic-html were \(82.6ms\) an \(80.6ms\) , with response time violation rates equal to \(7.3\%\) and \(4.6\%\) , respectively. Autoscaling was performed similarly also for the other three functions with equal desired response time (i.e., graph-bfs, graph-mst, thumbnailer). Compared with NEPTUNE, HPA achieved slightly worse performance when dealing with these functions. HPA was able to ensure a slightly lower average response time than NEPTUNE: short-running functions achieved an average response time of \(88.9ms\) , and of \(95.5ms\) with NEPTUNE. HPA reported higher variance (i.e., the average standard deviation for short-running functions was \(6.1ms\) for HPA and \(1.0ms\) for NEPTUNE). Then NEPTUNE was able to guarantee more stable response times over time. In fact, despite a higher average response time, the violation rate was equal to \(3.0\%\) with NEPTUNE and to \(3.3\%\) with HPA. This behaviour is mainly due to the fact that HPA employs a horizontal scaling approach, which means that it can either create a new replica or delete an existing one. Since each replica has a certain amount of resources assigned to it, HPA can allocate (or deallocate) a fixed amount of resources (usually in the order hundreds of millicores), while NEPTUNE can perform a finer-grained resource allocation (in the order of single millicores). For this reason, when it comes to allocating the right amount of resources, HPA is less precise than NEPTUNE. The observation on resource allocation further confirms the suboptimality of HPA. HPA allocated 4.35 cores on average while NEPTUNE only allocated 1.82 cores ( \(59.2\%\) fewer allocated resources).

The last row of the table refers to function video-processing. This function is characterized by being a long-running function since, unlike the other functions that take a few hundred milliseconds, it requires some seconds to compute the response. To cope with the higher response time, we employed higher integral and proportional gains (i.e., \(g_{int}=25000\) and \(g_{prop}=50000\) ). The average response time achieved by NEPTUNE was \(10.1s\) , which was very close to the target one (i.e., 10s), and all requests were served within the desired response time \(RT^R\) .

Table 4 shows how the performance of function video-processing changed when we used different integral and proportional gains. Note that when the gains of the PI controllers were too low (i.e., k = \(10^0\) and k = \(10^1\) ), the controllers could not allocate enough resources to cope with the workload. The average response times were \(26.2s\) and \(18.9s\) , in the two cases, and about half of the requests exceeded the desired response time.

On the other hand, when the gains were very high (i.e., k = \(10^4\) and k = \(10^5\) ), the amount of allocated resources was unstable. In fact, despite having acceptable average response times (i.e., \(10.9s\) and \(12.7s\) , respectively), the standard deviations of these runs were higher than the other configurations. The instability led to higher response time violations (i.e., \(2.2\%\) and \(12.7\%\) of the requests were not served within \(RT^R\) ) and resource over-provisioning (i.e., 17.5 and 18.3 cores, in the two cases).

The experiments showed that NEPTUNE can achieve very good performance if the integral and proportional gains were correctly set, and they highlighted the importance of proper configurations. When we used \(k=10^2\) and \(k=10^3\) , NEPTUNE was able to allocate resources to let no requests exceed the desired response time. Furthermore, the average response times were close to the target one of PI controllers (i.e., 10s) and the standard deviations were low. This highlighted the precise and robust control of NEPTUNE.

The second set of experiments aimed to assess the Contention Manager proposed at Node level when dealing with multiple applications at the same time. We used the same infrastructure described in Section 5.2 (i.e., master and worker nodes on a c5.2xlarge and a c5.9xlarge virtual machines, respectively). We also used a sinusoidal workload configured with \(MinUsers=1\) , \(MaxUsers=250\) , \(StartTime=60s\) , \(Period=1200s\) , and \(Duration=3600s\) , but instead of running one function at a time, we ran multiple functions concurrently. The workload was designed to have three peaks (at seconds 700, 1900, and 3100), and its goal was to consume all available resources. Then, we grouped the functions into two sets. SetA comprised four functions: graph-bfs, dynamic-html, thumbnailer, and video-processing. The workload was split as follows: \(1\%\) to video-processing and \(33\%\) to each other function. SetB comprised three functions: graph-mst and compression, video-processing; \(1\%\) of the workload again was assigned to video-processing and the rest was split equally between the other two functions (i.e., \(49.5\%\) to each function). Note that we decided to forward a significantly smaller percentage of requests to video-processing since it requires more resources per request.

Figure 5 presents one (randomly chosen) run of the experiments. Figures 5(a) and 5(b) (5(c) and 5(d)) illustrate the desired resources (i.e., the resources that should be allocated if nodes had infinite resources) and actual allocated resources (over time) for functions in SetA (SetB). The figures show the benefits of resource contention. In correspondence to workload peaks, allocated resources are similar to desired ones. This behavior is due to the proportional policy employed by the Contention Manager, which assigns resources in a fair manner (i.e., NEPTUNE splits resources according to the demand). We can also observe that desired resources only exceeded a few times the maximum amount of available resources (i.e., 36 cores), which means that NEPTUNE suggests resource allocations that, in the worst case, are close to feasible.

For comparison, we ran the same experiments with HPA, an autoscaling solution that does not implement any mechanism to solve resource contentions. HPA assign resources by using a FIFO (First In First Out) policy, that is, resources are assigned to the first function that asks for them. Figure 6 shows obtained results. SetA functions did not have a fair resource allocation compared with their demand. During workload peaks, the amount of desired resources for function video-processing (Figure 6(a)), was very similar to that of function graph-bfs, but the amount of allocated resources (Figure 6(b)) was significantly smaller. The second and third workload peaks, at seconds 1900 and 3100, respectively, highlight this behavior: less than 10 cores were allocated to video-processing and more than 20 to graph-bfs. Figures 6(c) and 6(d) exemplify a similar behaviour with SetB functions.

Autoscaling approaches—with no contention mechanisms—might propose unfeasible resource allocations. This severely degrades the effectiveness of these solutions. For example, after some 1900 seconds, HPA would like to have 60 cores for SetA functions, while only 36 cores were available.

The third set of experiments focused on GPU management for computationally intensive functions. The experiments were conducted in a heterogeneous environment using two c5.xlarge virtual machines and one g4dn.xlarge virtual machine that was equipped with a GPU. The functions used for the experiments were resnet-a and resnet-b, both embed the ResNet neural network in inference mode and were configured to exploit the available GPU. For each run, we use a ramp workload with \(MinUsers=10\) , \(MaxUsers=30\) , \(StartTime=10s\) , \(RampTime=30s\) , and \(Duration=1200s\) .

Figure 7 shows the results of a single execution of the experiments since the applications behaved similarly in all the runs. In addition, Figure 7(a) only presents the average response time of the GPU and of the CPUs with resnet-a since the two applications showed a similar trend.

The workloads executed on the GPU produced an almost constant response time and never violated the threshold. Initially, the incoming workload was small, and the routing policies redirected the traffic to the GPU. It handled the whole incoming workload till its full utilization, for some 50 seconds. At that point, the Community level quickly reacted by updating the routing policies, and part of the workload was forwarded to the CPUs. The response time shows an initial peak and a few violations caused by the cold start of the functions. Then, the Node level came into play and allocated the CPUs to bring the response time close to the set point.

Figure 7(b) reports the average response time for both resnet-a and resnet-b. It contains the average response time when functions are running on GPUs, on CPUs, and the aggregated results. The GPU kept the response time almost constant and caused no violations. In particular, the average response times were 180ms and 183ms for resnet-a and resnet-b, three times smaller than the threshold. The use of CPUs led to a wider distribution. In fact, the standard deviations of CPU instances were \(42.3ms\) and \(66.1ms\) for resnet-a and resnet-b, respectively. PI controllers managed the CPU cores: they introduced an initial transient period needed to adjust the initial core allocation.

The CPU-only replicas of resnet-a and resnet-b served 98.3% and 100%, respectively, of the requests within the set response time. The GPU handled 70% of the requests and the CPUs handled the rest. The total number of violations in both cases is close to 0. The results also say that the transparent GPU management provided by NEPTUNE allowed the two functions (resnet-a and resnet-b) to seamlessly share the same GPU. Figure 7(b) shows that the sharing did not degrade the performance of either application since both applications presented similar distributions of response times. Moreover, the interplay between GPU and CPU instances helps keep the average response time as low as \(200.3ms\) for resnet-a and \(202.2ms\) for resnet-b.

The fourth set of experiments aimed to evaluate the placement and routing solution proposed in NEPTUNE. Specifically, the objective was to evaluate different configurations ( \(\alpha \in \lbrace 0, 0.5, 1\rbrace\) ) of the Community level and analyze the tradeoff between minimizing the network delay and the number of MEC nodes needed to manage the incoming workload. The experiments were conducted using a multi-region environment with four m5.xlarge virtual machines placed across four different areas (i.e., Area A, B, C, and D). The experiments employed the functions provided in SeBS and we used the three different workloads (i.e., \(W_{cs}\) , \(W_{td}\) , \(W_{tel}\) ) generated from real-world traces and with a duration of 30 minutes. The workload takes into consideration that function video-processing is very resource demanding, only \(1\%\) of the users were designed to request that function, while the remaining users were randomly spread across all other functions (i.e., \(19.8\%\) to each function).

Table 5 reports the results (mean and standard deviation are reported as “ \(\mu \pm \sigma\) ”) of the experiments with workload \(W_{cs}\) . Concerning network delay and \(\alpha =0\) , NEPTUNE aimed only to place function instances close to the incoming workload. In fact, we can observe that under this configuration, NEPTUNE achieved the lowest network delays: from 17.1ms (function compression) to 199.1ms (function graph-mst). Increasing \(\alpha\) made NEPTUNE also consider the cost factor of running the MEC infrastructure but also led to higher network delay. For example, with \(\alpha =0.5\) and \(\alpha =1\) , the average times spent by requests in the network for function graph-mst were, respectively, 265.5ms and 341.2ms (more than the required response time \(RT^R_f=200\) ms).

Higher network delays also led to higher response time violations. For instance, with \(\alpha =0\) , only \(1.5\%\) of the requests to function compression exceeded the required response time, while with \(\alpha =0.5\) and \(\alpha =1\) , the violation rates were, respectively, 4.8 and 5.9 times higher. Regarding allocated cores to functions, there were no statistically significant differences between the three configurations. This means that, on average, the three configurations of NEPTUNE allocated a similar overall amount of cores to functions. However, if we consider the amount of nodes used to serve the incoming workload, we can observe that with \(\alpha =1\) , NEPTUNE activated on average only 1.3 nodes, while configurations \(\alpha =0\) and \(\alpha =0.5\) required 3 and 2.5 times more active nodes.

In comparison with competitor approaches, results show that NEPTUNE configured with \(\alpha =0.5\) was able to outperform VSVBP, MCF, and CR-EUA in most aspects. For instance, the average network delay of NEPTUNE with \(\alpha =0.5\) was 113.8ms, while VSVBP, MCF, and CR-EUA generated, respectively, \(43.7\%\) , \(20.7\%\) , and \(2.5\%\) higher network delays. Competitors also served fewer requests within the desired response time ( \(19.7\%\) , \(9.9\%\) , and \(12.3\%\) more violations) while allocating more resources ( \(53.5\%\) , \(40.3\%\) , and \(53.3\%\) more). However, from the cost point of view, the competitors required on average only 1.6, 1.4, and 2.7 active nodes to handle the incoming workload, while NEPTUNE with \(\alpha =0.5\) required 3.2 nodes.

This behavior can be explained by the fact that competitors model the problem as a user allocation problem and aim to maximize the number of users allocated to as few MEC nodes as possible. However, they do not take into consideration some aspects of real-world workloads. For instance, they do not consider the scenario where new users appear at runtime or existing users move across different areas. Furthermore, the absence of mechanisms that minimize service disruption also contributed to higher network delays and violation rates.

Table 6 reports an overall view of the results of NEPTUNE and competitors when we used the three different workloads ( \(W_{cs}\) , \(W_{td}\) and \(W_{tel}\) ). The results confirm what was observed with workload \(W_{cs}\) . For instance, if we consider the three configurations of NEPTUNE, the best network delays, and response time violations, are obtained with \(\alpha =0\) in almost all cases. Furthermore, obtained allocated cores are similar across the three \(\alpha\) configurations, while only with \(\alpha =1\) the number of active nodes was reduced.

Compared with competitors, NEPTUNE obtained a more reliable behaviour. For example, with workload \(W_{td}\) , which resulted to be the most difficult workload to handle, NEPTUNE with \(\alpha =0.5\) exceeded the required response time only \(30.5\%\) of the time, while VSVBP, MCF, and CR-EUA obtained violation rates equal to \(52.0\%\) , \(35.3\%\) , and \(60.2\%\) , respectively. Overall, NEPTUNE obtained \(44.5\%\) lower network delays, \(42.5\%\) less response time violations while allocating \(50.8\%\) fewer resources with respect to its competitors.

We conducted a large set of experiments and showed that NEPTUNE can minimize network delay, reduce response times, efficiently allocate resources, solve resource contentions, and manage CPUs and GPUs transparently. However, we must highlight some threats that may constrain the validity of obtained results [60]:

Internal Threats. The experiments were conducted on a cloud infrastructure, which allowed us to emulate significant latencies among nodes by utilizing different regions and availability zones. While this approach provided valuable insights into the performance of the proposed system, it is essential to consider the potential impact of other aspects that can exist in real-world MEC infrastructures. For example, in cloud environments network performance is generally not subject to unforeseen fluctuations. Conversely, at the edge, the network conditions could be more dynamic and less reliable. To mitigate these kind of issues, our theoretical framework and prototype have been designed to take into account most of such dynamic conditions. For example, the inter-delay among nodes is constantly measured and updated, thus decisions are taken without any assumptions on (nearly) constant network performance. Moreover, in our experiments, we used real-world workloads that capture the characteristics of edge use cases (e.g., taxis moving around different areas). Additionally, the functions we took from the literature do not provide a recommended value for the response time \(RT^R_f\) . To obtain this value, we used an iterative process. We started from 50ms, and we incremented the value by 50ms until the function was able to serve at least 50% of requests in a time equal to \(RT^R_f/2\) .

External Threats. Some of our assumptions may limit the generalization of the experiments. Since NEPTUNE adopts the serverless paradigm, it assumes that functions be either stateless or depend on an external database. Currently, NEPTUNE only partially models database interactions. The delays introduced by reading from and writing to a database are modeled at the Node level as a non-controllable stationary disturbance of the response time (e.g., a Gaussian noise). We previously demonstrated [4] that NEPTUNE can efficiently handle functions that rely on a database with a precision that is similar to those without data dependencies.

Moreover, our approach has only been tested on a single cloud provider (AWS) posing a potential limitation to the generalizability of our study. However, we have designed NEPTUNE to be cloud-agnostic, not relying on AWS-specific features. Thus, we expect that similar performance could be obtained across different cloud providers.

Finally, the assumption that GPUs should be prioritized over CPUs may not be valid in scenarios where cost is determined by electricity or battery availability. In such instances, a careful evaluation is needed between the computational speed-up of GPUs and their associated higher costs. However, these objectives are beyond the scope of this article and will be addressed in our future work.

Construct and Conclusion Threats. We adopted a special-purpose monitoring infrastructure to gather metrics during the experiments. Such infrastructure collects data about latency and resource allocation. However, those data can be subject to uncertainty and noise, which could compromise the measurement process. To demonstrate the validity of our claims, and reduce uncertainty and noise, we repeated all experiments five times, and obtained results are statistically robust.