As the workloads of jobs change dynamically, the optimal resource partitioning configuration for each colocation would vary over time. In order to achieve the optimal system throughput consistently, we need an adaptive resource partitioning strategy which can tune resource partitioning configuration dynamically according to the workload variations. To address this issue, we leverage deep reinforcement learning (DRL) to propose an end-to-end resource partitioning strategy. The DRL-based resource partitioning approach is very efficient, which directly generates the optimal resource partitioning decision according to the real-time system state. Moreover, the DRL-based approach naturally has generalization ability, making it highly robust to dynamic workload changes.
4.2.1 Background of Deep Reinforcement Learning.
Reinforcement Learning (RL) is a kind of approach in which an agent interacts with the environment and learns an optimal policy by trial and error. At each time step t, the agent receives a state \(S_t\) and selects an action \(A_t\) according to a policy \(\pi (A_t|S_t)\) . The policy is the strategy that the agent employs to determine the next action based on the current state, which is a mapping from state \(S_t\) to actions \(A_t\) . After the action is performed, the agent receives a scalar reward \(w_t\) , and the environment transitions to the next state \(S_{t+1}\) . The reward is the feedback reflecting how good is the action \(A_t\) in the state \(S_t\) (a larger reward indicates a better action). This process continues until the agent finds the optimal policy, which maximizes a discounted and accumulated reward \(\sum _{t=0}^{\infty } \gamma ^t w_{t}\) ( \(\gamma \in (0,1]\) ) is the discount factor). Finding the optimal policy is crucial for RL, but is not an easy task. A recently popular approach is to use deep neural network (DNN) to approximate the policy, which we call Deep Reinforcement Learning (DRL). This is because DNN has strong fitting ability to establish the mapping from input to output without relying on domain knowledge.
We would like to say that DRL is suitable for resource partitioning. Once the DRL model is well-trained, the partitioning decision can be made very quickly, because the inference of neural network is fast. A well-trained neural network has natural generalization to the inputs which have not appeared in the training scenario, in our context, it can adapt to unseen workloads, colocations and jobs, making it highly robust to dynamic system state changes.
4.2.2 The DRL Model for Resource Partitioning.
A DRL model consists of the following concepts: environment, action, state, reward, and policy. In the context of resource partitioning, the concepts are defined as follows:
Environment. The DRL model works on a single server, for partitioning the shared resources among the jobs colocated on the server. So, we define the server as the environment.
State. The state is defined as the server state in our context, which is also the input of the DRL model. How to define the server state is non-trivial, because (1) the server state must be able to reflect the workload characteristics of jobs and resource contention behaviors across jobs; and (2) the server state must be easy to collect since the resource partitioning decision should be generated quickly. The server state in our DRL model is composed by three parts: performance counters associated with jobs, the current partitioning configuration, and the number of threads of jobs. We include the number of threads of each job in the system state, because multi-threaded jobs are sensitive to the number of CPU cores allocated.
Consider the i-th server in the datacenter. Denote by \(c_i\) the set of jobs colocated on the i-th server. We use \(PMC_{i,t}=(pmc_{i,t}^1, \ldots , pmc_{i,t}^{|c_i|})\) to denote the performance counters for the jobs in \(c_i\) at time step t, where \(pmc_{i,t}^j\) refers to the performance counters associated with the j-th job. We use \(p_{i,t}=\lbrace p_{i,t}^1,\ldots , p_{i,t}^{|c|}\rbrace\) to denote the resource partitioning configuration of \(c_i\) at time step t, where \(p_{i,t}^j\) refers to the allocation of resources for the j-th job. So, the server state at time step t can be represented by \(S_{i,t} = \lbrace PMC_{i,t}, D_i, p_{i,t}\rbrace\) , where \(D_i\) denotes the numbers of threads of jobs.
Action. The action is the output of DRL model, which refers to the resource partitioning decision in our context. For the i-the server, we use \(A_{i,t} = \lbrace a_{i,t}^1, \ldots , a_{i,t}^{|c|}\rbrace\) to denote the action at time step t, where \(a_{i,t}^j\) refers to the resource allocation for the j-th job.
Reward. The goal of resource partitioning is to maximize the system throughput. Let \(S_{i,t}\) denote state of the i-th server at time t. Suppose an action \(A_{i,t}\) (i.e., a new resource partitioning configuration) produced by the DRL model is applied to the i-th server at time t. Let \(S_{i,t+1}\) denote the state of the i-th server after \(A_{i,t}\) is applied. The reward \(w_{i,t}\) of the DRL model for \(A_{i,t}\) is defined as the server throughput in state \(S_{i,t+1}\) .
Policy. The DRL method uses DNN to approximate the policy model. The policy model maps the input (i.e., the server state) to the output (i.e., the partitioning decision). Figure
3 shows the design of the DRL policy model. Overall, the policy model employs an encoder-decoder architecture. The encoder first transforms the input to a high-dimensional vector through a stack of
fully connected (fc) layers. Then, the transformed features are fed to the decoder network through an attention layer. The decoder network adopts the
Gated Recurrent Unit (GRU) to generate the partitioning decision. The GRU architecture is commonly used for sequential decision. In our context, it determines the resource allocation for the jobs one by one in a sequential manner.
4.2.3 Reward Prediction for Reducing Training Overhead.
DRL typically requires a large number of interactions between agent and environment to train the model. In the context of resource partitioning, a single interaction may take over 10 seconds, because we need to generate a new partitioning configuration according to the partially-trained DRL model, apply the new generated configuration to real system, observe the reward, and finally update the DRL model using the reward. Since the DRL model needs to be trained by tens of millions of interactions, the total training time is about \(O(10^5)\) hours, which is not feasible even if the training process is one-shot. So, a crucial issue of applying DRL to resource partitioning is how to reduce the training overhead.
To address this issue, we propose a prediction model to estimate the reward of a given action, without interacting with the real system. Suppose the i-th server is in state \(S_i = \lbrace PMC_i, D_i, p_i\rbrace\) at some time point, where \(PMC_i\) denotes the performance counters of jobs, \(p_i\) denotes the current resource partitioning configuration, and \(D_i\) denotes the numbers of threads of jobs. Given an action \(A_i\) (i.e., a resource partitioning decision), let \(IPC_{i}^j\) denote the IPC of the j-th job in \(c_i\) if \(A_i\) is applied to \(S_i\) . The prediction model takes \((S_i, A_i)\) as input to estimate \((IPC_i^1, IPC_{i}^2,\ldots , IPC_{i}^{|c_i|})\) , without applying \(A_i\) to the real system. The throughput of the colocation is defined as the total IPC of all the jobs, which can be computed by \(\sum _{j=1}^{|c_i|} IPC_{i}^j\) .
We leverage DNN to build the resource partitioning reward prediction model. Briefly, an fc layer is used to encode the input features at the current time step, and a multi-head attention module is used to synthesize the outputs of the fc layer. The multi-head attention module is followed by another fc layer, which generates the final output, i.e., the IPCs of the jobs at the next time step.
Training the reward prediction model still needs to interact with real systems to collect training samples. However, as the number of training samples required for training the prediction model is much less than that for training the DRL model, the prediction model can significantly reduce the training time of the DRL model (see details in Section
5.5.7).