Pareto Optimization of Analog Circuits Using Reinforcement Learning

In the past, some works have studied the methods of generating the Pareto set by both analytical and data-driven methods. Analytical methods, [4], use line search approaches that depend on gradient information to find the Pareto set. However, increasing the dimensionality of the objective space limits an analytically solvable solution to the line search. Another popular method used is NSGA-II [3], which utilizes a genetic algorithm approach. This algorithm introduces a fast approach to find the dominance depth of each sample point in the population. After sampling an initial population of the design space, the genetic algorithm searches the design space using crossover and mutations. There is a fixed budget on the number of function evaluations, and the algorithm ends as this budget is reached. By carefully adjusting the crossover and mutations, the algorithm is able to achieve exploration of the state space.

Reference [15] utilizes Gaussian Processes and the Bayesian framework to query for points in the design space. Here, Pareto set generation is accomplished by iterative search and dominance depth assignment to the points sampled. The work in Reference [9] employs search over the objective space by using a combination of genetic algorithm and Bayesian optimization. The Gaussian process regression framework is used to provide a surrogate model that is then used for pre-selecting Pareto dominant points using genetic algorithms. However, Reference [6] proposes analog circuit sizing in two phases, alternating between Bayesian optimization and evolutionary Pareto front search using similar approaches to Reference [9] while introducing constraints on performance specifications and parasitics. Yet another work that uses Bayesian optimization for multi-objective optimization is Reference [14]. The authors of Reference [14] propose to reduce the time complexity of matrix inversion from \(\mathcal {O}(n^3)\) to \(\mathcal {O}(n^2)\) using an incremental learning technique and uses a modified acquisition function matrix suited for multi-objective optimization. Reference [13] use similar self-adaptive incremental learning techniques to Reference [14] to use surrogate approximators for pre-selection of valid Pareto optima; design points for evolutionary algorithms to simulate. Reference [7] also use Bayesian optimization for LDE aware analog circuit sizing. Our proposed method is based on complete neural network implementations. The time complexity of operation of our method is independent of the number of samples collected unlike any of the Bayesian optimization-based approaches, which makes our method a lightweight optimization engine. Second, the predictive ability of the proposed method to generate new Pareto optimal points along a user specified preference direction during inference time makes our method tunable to specific user preferences during test time.

Reinforcement learning recently has been used in single-objective optimization or composite multi-objective optimization in circuit design in References [2, 8, 10, 11]. These works predominantly use off-policy RL algorithms under continuous action spaces. However, they do not consider different tradeoffs in design objectives to form a Pareto set. Another work that closely resembles our work is Reference [5]. Reference [5] proposes to perform single-objective optimization using a set of static weights to weight each objective and use this as a reward function in the reinforcement learning framework. However, this approach does not explicitly capture tradeoffs between different objectives in the optimization process. Thus our work is different from Reference [5] on three main algorithmic fronts.

(i) We use vectorized rewards instead of scalar rewards to specify the exact values of the objectives being optimized (rewards are not scalarized—for example \(\alpha _1\) .Gain + \(\alpha _2\) .UGF). (ii) We have a dynamically changing preference direction to weight the objectives. This is unlike fixed weights that Reference [5] uses. This dynamically changing preference direction G is used to find the Pareto optimal points on the Pareto front, which is a capability not explored by Reference [5]. (iii) We propose a vectorized version of the DDPG algorithm where the Q-function is itself vectorized to perform multi-objective optimization. This is unlike Reference [5] where Q values are scalars and do not consider tradeoffs between multiple objectives.

In the field of multi-objective RL (MORL) in general, the work in Reference [12] introduces methods to query for the Pareto set. But these approaches in MORL, deal with scenarios where the actions are discrete and countably few. For analog circuit sizing, although sizing solutions are typically on a grid, the number of possible values on the grid might be many in number and can be thought of as continuous. While using MORL for such problems, it calls for designing RL algorithms that operate with continuous actions instead of just discrete ones.

The goal in multi-objective optimization is to find a set of optimal solutions—the Pareto set. The solutions of the Pareto set do not dominate any other solution in the same set. A point \(\mathbf {s_1}\) dominates another point \(\mathbf {s_2}\) if \(\forall i \in \lbrace 1,2,\ldots m\rbrace\) , \(f_i(\mathbf {s_1}) \ge f_i(\mathbf {s_2})\) , and \(\exists j \in \lbrace 1,2, \ldots , m\rbrace\) such that \(f_j(\mathbf {s_1}) \gt f_i(\mathbf {s_2})\) . Thus, we say \((\mathbf {s_1} \succ \mathbf {s_2})\) . With the notion of dominance established, we want to find all the points \(\lbrace \mathbf {s} \rbrace\) that are not dominated by any other point. The set of these points is called the Pareto set \(\mathcal {S}\) . Let us say that we are optimizing for multiple objectives given aswhere \(\mathbf {s}\in \mathbb {S} \subset \mathbb {R}^D\) and m is the number of objectives to be optimized and D is the dimensionality of the design space. Considering a maximization objective, our aim is to find

The set of solutions \(\mathbf {s} \in \mathcal {S}\) , for the above equation belongs to the Pareto set and the resulting metric or figure of merit (FOM) values F, constitutes the Pareto front \(\mathcal {P}\) .

Next we briefly introduce some important concepts used in this work.

To apply reinforcement learning to solve multi-objective optimization problems, we need to specify the internals of the RL agent. Keeping in mind the necessity to optimize for m objectives, we define the concept of a goal vector defined aswhere \([g_i]_{i=1}^{m}\) is the preference or the weight associated with each objective to be optimized. The realization of the goal vector \(\mathbf {G} \in \mathcal {G}\) , is such that \(\mathcal {G} \subseteq \lbrace 0,1\rbrace ^m\) andwhere \(i = [1,2, \ldots m]\) .

Thus, the goal vector specifies the designer’s intent to trade off among multiple competing objective values and specifies the direction of the search in the m-dimensional objective space. We next look at how the goal vector is included in the reinforcement learning flow. We first define the notion of the state vector \(\mathbf {s}\) , which specifies the sizing solutions for the analog circuit,

whereW and L stand for the width and length of transistors respectively. We enforce the state space to be bounded within a certain normalized range, i.e., constrain each entry of the state vector \(a\lt \mathbf {s}_i\lt b\) to adhere to the operating technology. We illustrate this in Figure 1 in the constraint-checker module.

To integrate the goal vector into the RL state, we simply concatenate the goal vector with the \(\mathbf {s}\) and define it as the goal-state of the system \(\mathbf {s}_G\) ,

The goal-state of the system specifies the current sizing solution in addition to also giving information about the designer’s intent or preference \(\mathbf {G}\) . The goal-state can be thought of as an extension to the state space. The same sizing solution \(\mathbf {s}\) could differ, based on the specified goal state \(\mathbf {G}\) , in the state space according to designer’s preference.

We define the action a, taken by the policy network of the RL agent, as the incremental change in the sizing solutions of the state vector \(\mathbf {s}\) . Actions are sampled from the set of actions \(\mathcal {A}\) that consist of continuous values. Although sizing in analog circuits is on a grid, the precision change in the sizing ( \(\delta L_i, \delta W_i\) - lengths and widths) can be thought of as continuous valued. This is also similar to the modelling choice chosen by Reference [2] for the definition of actions,

Unlike the traditional RL framework, the reward in our work also targets at multi-objective optimization and thus is also m-dimensional. The reward vector has the same dimensionality as the number of objectives being maximized. The reward can be any of the FOM of the circuit performance that are competing. These FOMs are normalized between certain ranges, usually between 0 and +1,

In Q-learning, the Q value is a measure of goodness of a given state–action pair \((\mathbf {s},a)\) . The agent updates its estimate of this goodness through the Bellman equation given aswhere \(r(.)\) is the reward and \(\mathbf {s}^{\prime }\) and \(a^{\prime }\) indicate the next state and action seen in the RL trajectory.

As indicated in the work in Reference [12], we see that the Q function, in the multi-objective setting, unlike single-objective optimization, is also m dimensional. The Q value is a measure of goodness of a given goal-state–action pair \((\mathbf {s}_G,a)\) . The inputs to the Q network are the goal-state \(\mathbf {s}_G\) and the action a. The Bellman operator in the multi-objective setting is given bywhere the filter \(\mathcal {H}\) is called the Bellman optimality filter and \(P(\cdot \mid \mathbf {s},a)\) is the transition probability and \(\mathbf {s^{\prime }}\) is the next state seen in the RL trajectory,

The optimality filter provides \((\mathrm{arg}_{Q})\) , the Q value that maximizes the dot product \(\mathbf {G}^T\cdot Q(\mathbf {s},G^{\prime },a)\) . In other words, it finds the Q value in alignment with the preference \(\mathbf {G}\) . In the single-objective case, this just reduces to finding the Q value for the best action, i.e., \({\mathrm{max}}_{a^{\prime } \in \mathcal {A}} Q(\mathbf {s},a^{\prime })\) . Thus, Equation (11) is a general form of the single-objective case. The authors of Reference [12] elucidate that the optimality operator \(\mathcal {T}\) is a \(\gamma\) -contraction under a certain distance metric. The authors prove through the Banach fixed point theorem that the operator \(\mathcal {T}\) has a fixed point, meaning that repeated application of the operator on the multi-objective Q function leads to convergence of the Q function. Figure 2 gives a visual intuition of the Q updates.

We propose an actor network that is a parameterized network that approximates the best action to take, given the current sizing solution \(\mathbf {s}\) and the designer’s preference \(\mathbf {G}\) . In other words, the actor policy actively tries to push the next query sizing solution \(\mathbf {s^{\prime }}\) in the direction of optimizing the objectives according to the designer’s preference \(\mathbf {G}\) . The actor policy is parameterized by \(\phi\) ,

For a given preference level \(\mathbf {G}\) specified by the user, the preference policy suggests the most optimal goal-state that needs to be prescribed to maximize the dot product Q network output and the user preference \(\mathbf {G}\) . Both the preference policy and the actor policy are part of the optimality filter \(\mathcal {H}\) . The preference policy is parameterized by \(\omega\) ,

The preference policy and the actor policy networks suggest the optimal search direction and the incremental change in sizing so that the Q value vector is aligned with the user preference \(\mathbf {G}\) . Next, we look at how the RL agent is trained.

The training phase entails the convergence of the critic, actor, and the preference networks in accordance with the Bellman optimality filter. The estimated value that the current critic needs to converge to, for a goal-state–action pair \((\mathbf {s}_G,a)\) , is as follows:

We define the critic loss as a distance minimization on a distance operator \(\mathcal {D}\) between the current Q vector and the estimated \(Q_{\mathrm{est}}\) vector,

We used \(\mathcal {D}\) as a \(\mathrm{L}_2\) norm of the Q vector value difference, i.e., \(\mathcal {D}(Q,Q_{est}) = ||Q-Q_{est} ||_{2}^2\) . The policy to drive the right actions and preferences is trained by maximizing the critic’s output for a given preference level \(\mathbf {G}\) as shown in Equation (12),

The policy loss can be seen as driving the actor and preference policies to output values that drive the Q function to align with the given preference \(\mathbf {G}\) . The samples collected through the training process reflect the RL agent’s ability to intelligently explore different regions of the design space (space of sizing solutions), while being aware of the tradeoffs in the objectives it tries to optimize for. When new objective values are received through training, we collect them and pass them through a Pareto front identifying filter function, which takes all data points (composed of objective values) \(\mathrm{F} = \lbrace f_1, f_2, \ldots , f_m\rbrace\) and outputs a Pareto front. The resulting Pareto Front is the result of training the agent and comprises a certain number of points in it. Note that the set of these Pareto front points is static, meaning that they are a fixed set of points that are identified during the training phase,where T is the number of training steps. We show the training procedure for continuous Pareto optimization for analog circuits in Algorithm 1.

The RL approach to Pareto optimization uses neural networks as function approximators to predict the next action to take given a sizing solution. This way, the trained RL agent has an in-built knowledge of the state transition from one sizing solution to the next, in the circuit simulator. We can thereby exploit the trained agent to query for different user preferences \(\mathbf {G}_{\mathrm{U}}\) and augment the points on the Pareto front \(\mathcal {P}\) generated during training. This has the benefit of adding additional points on the Pareto front that are not seen during training.

We know that the actor policy network takes into account the designer’s preference \(\mathbf {G}\) along with the state \(\mathbf {s}\) , i.e., \(\mu (\mathbf {s}, \mathbf {G}, \phi)\) . The trick here is to initialize any random state \(\mathbf {s}\) along with the designer’s particular preference \(\mathbf {G}_{\mathrm{U}}\) and let the trained RL agent act according to its actor policy for a few steps. The policy forces the RL agent to take steps in the direction of the preference vector \(\mathbf {G}_{\mathrm{U}}\) . Thus, this trained model approach can dynamically output design points according to the designer’s preference by exploiting the predictive power of the RL agent. We demonstrate this in Algorithm 2.

To update the critic, actor and the preference losses, we use mini-batch optimization. The mini-batch is sampled from the replay buffer. The batch size is set as B. Also, for each entry in the batch, we randomly sample Z preference directions \(\mathbf {G}_Z\) . The losses are averaged over the preferences \(\mathbf {G}_Z\) first, for each entry in the batch and then over all B entries. We show the RL algorithm flow in Figure 1.

The two-stage differential amplifier is shown in Figure 3. The circuit consists of a 14-parameter search space, which includes sizing solutions of the circuits along with the resistor and capacitor values.

As can be seen in Figure 4(a), our RL method during training (RL-train), visually generates a Pareto front of a superior quality in comparison with NSGA-II, BO, and Monte Carlo sampling. Table 1, enlists the hypervolume along with the number of samples and the runtime for each algorithm. First, we see that RL-train shows the least hypervolume and thus performs better than all the three reference methods. Next, we see that Monte Carlo and BO take almost 6 \(\times\) the data consumed by RL-train but produce a Pareto front with a larger hypervolume that can also be visually inspected in Figure 4(a). Furthermore, due to the sample efficiency, simulation time is greatly reduced. A drawback of BO is that it has a complexity of \(\mathcal {O}(N^3)\) and is thus very slow. We believe that our RL-train method performs better than NSGA-II due to the fact that the RL agent understands how sizing affects both the gain and the bandwidth due to the predictive power of the Q network.

In Table 2, the first row demonstrates the fraction of points contributed by the Pareto front \(\mathcal {A}=\rm RL\) -train when combined with Pareto fronts \(\mathcal {B}=\rm NSGA\) -II, BO, and Monte Carlo. As can be seen RL-train has \(\mathcal {FC}\) values greater than 0.5 for all the three comparison methods, which means that RL-train is the dominant contributor to the combination Pareto front.

We next experiment with the folded cascode amplifier. The amplifier has 18 parameters in its search space. As seen in Table 3 the RL-train has the least hypervolume among all three methods and thus has a better quality Pareto front. It also consumes the least amount of data and has the least simulation time. Table 4 indicates again that the RL-train is a dominant contributor to the combination Pareto fronts.

The hysteresis comparator has a 12-parameter design space and presents a more complex Pareto front. However, it can be seen that the RL-train method we propose is able to find a better Pareto front. As in the other circuits, we see the best hypervolume and also a best fractional contribution by RL-train.

As mentioned before, we can use the predictive power of the trained RL agent to verify if more points can be populated on the Pareto front. To do so, we sweep through the possible values of user preferences \(\mathbf {G}_U\) monotonically for some random state initialization near the current Pareto front \(\mathcal {P}\) . We record the number of time steps required to reach a solution point that dominates points in the current Pareto front \(\mathcal {P}\) . We do this over N state initializations and note how efficient the search process is, in finding the dominating Pareto front point. We limit the number of steps in each RL run (as in Algorithm 2) in inference to \(L=10\) . We define \(x_{U,i}\) to be the minimum number of steps taken to reach a Pareto-dominant point over all sweeps \(\mathbf {G}_U\) for a given state initialization i. The new Pareto point prediction capability of the trained agent is high if \(x_{U,i} \rightarrow 0\) . In short we define the efficiency of the trained agent as

From Table 7, all the trained models exhibit good prediction capabilities for the dynamic user preference \(\mathbf {G}_U\) , which demonstrates that the trained RL agent can be saved to query for Pareto front points not seen during training.