Manas: multi-agent neural architecture search

Abstract

The Neural Architecture Search (NAS) problem is typically formulated as a graph search problem where the goal is to learn the optimal operations over edges in order to maximize a graph-level global objective. Due to the large architecture parameter space, efficiency is a key bottleneck preventing NAS from its practical use. In this work, we address the issue by framing NAS as a multi-agent problem where agents control a subset of the network and coordinate to reach optimal architectures. We provide two distinct lightweight implementations, with reduced memory requirements (1/8th of state-of-the-art), and performances above those of much more computationally expensive methods. Theoretically, we demonstrate vanishing regrets of the form ${\mathcal {O}}(\sqrt{T})$, with T being the total number of rounds. Finally, we perform experiments on CIFAR-10 and ImageNet, and aware that random search and random sampling are (often ignored) effective baselines, we conducted additional experiments on 3 alternative datasets, with complexity constraints, and 2 network configurations, and achieve competitive results in comparison with the baselines and other methods.

1 Introduction

Determining an optimal architecture is key to accurate deep neural networks (DNNs) with good generalisation properties (Szegedy et al., 2017; Huang et al., 2017; He et al., 2016; Han et al., 2017; Conneau et al., 2017; Merity et al., 2018). Neural architecture search (NAS), which has been formulated as a graph search problem, can potentially reduce the need for application-specific expert designers allowing for a wide-adoption of sophisticated networks in various industries. Zoph and Le (2017) presented the first modern algorithm automating structure design, and showed that resulting architectures can indeed outperform human-designed state-of-the-art convolutional networks (Ko, 2019; Liu et al., 2019). However, even in the current settings where flexibility is limited by expertly-designed search spaces, NAS problems are computationally very intensive with early methods requiring hundreds or thousands of GPU-days to discover state-of-the-art architectures (Zoph and Le, 2017; Real et al., 2017; Liu et al., 2018a, 2018b).

Researchers have used a wealth of techniques ranging from reinforcement learning, where a controller network is trained to sample promising architectures (Zoph and Le, 2017; Zoph et al., 2018; Pham et al., 2018; Bender et al., 2020), to evolutionary algorithms that evolve a population of networks for optimal DNN design (Real et al., 2018; Liu et al., 2018b; Lopes and Alexandre, 2022), to optimization on random graphs (Ru et al., 2020). Alas, these approaches are inefficient and can be extremely computationally and/or memory intensive as some require all tested architectures to be trained from scratch. Weight-sharing, introduced in ENAS (Pham et al., 2018), can alleviate this problem. Even so, these techniques cannot easily scale to large datasets, e.g., ImageNet, relying on human-defined heuristics for architecture transfer. Recently, low-fidelity estimates, performance predictors and guiding mechanisms have also been studied to improve the search cost and reduce the memory and computation required (Mellor et al., 2021; Lopes et al., 2021; White et al., 2021a; Ning et al., 2021; White et al., 2021b; Lopes et al., 2022). More, gradient-based frameworks enabled efficient solutions by introducing a continuous relaxation of the search space. For example, DARTS (Liu et al., 2019) uses this relaxation to optimise architecture parameters using gradient descent in a bi-level optimisation problem, while SNAS (Xie et al., 2019) updates architecture parameters and network weights under one generic loss. Still, due to memory constraints the search has to be performed on 8 cells, which are then stacked 20 times for the final architecture. This solution is a coarse approximation to the original problem as shown in Sect. 6 of this work and in Yang et al. (2020), Yu et al. (2019), Li and Talwalkar (2019), receiving some criticisms outlined in Zela et al. (2018), Yang et al. (2020), Wan et al. (2022). In fact, we show that searching directly over 20 cells leads to a reduction in test error (8% relative to Liu et al. (2019)). ProxylessNAS (Cai et al., 2019) is one exception, as it can search for the final models directly; nonetheless they still require twice the amount of memory used by our proposed algorithm, while offering no theoretical guarantees.

To enable the possibility of large-scale joint optimisation of deep architectures we contribute MANAS, the first multi-agent learning algorithm for neural architecture search. MANAS’ multi-agent framework is inspired by a multi-arm bandit setting (Auer et al., 1995), where here each agent is associated with one layer selection and a global network is optimised based on each agent decision. This class of algorithms has been widely explored to solve a range of problems (Bouneffouf et al., 2020), such as autonomous driving (Shalev-Shwartz et al., 2016), recommendation systems (Li et al., 2010), and active learning (Bouneffouf et al., 2014). Here we introduce the approach in neural architecture search.

Our algorithm combines the memory and computational efficiency of multi-agent systems, which is achieved through action coordination with the theoretical rigour of online machine learning, allowing us to balance exploration versus exploitation optimally. Due to its distributed nature, MANAS enables large-scale optimisation of deeper networks while learning different operations per cell. Theoretically, we demonstrate that MANAS implicitly coordinates learners to recover vanishing regrets, guaranteeing convergence. Empirically, we show that our method achieves state-of-the-art accuracy results among methods using the same evaluation protocol but with significant reductions in memory (1/8th of Liu et al. (2019)) and search time (70% of Liu et al. (2019)).

The multi-agent (MA) framework is inherently scalable and allows us to tackle an optimization problem that would be extremely challenging to solve efficiently otherwise: the search space of a single cell is $8^{14}$ and there is no fast way of learning the joint distribution, as needed by a single controller. More cells to learn exacerbates the problem, and this is why MA is required, as for each agent the size of the search space is always constant.

In short, our contributions can be summarised as: (1) framing NAS as a multi-agent learning problem (MANAS) where each agent supervises a subset of the network; agents coordinate through a credit assignment technique which infers the quality of each operation in the network, without suffering from the combinatorial explosion of potential solutions. (2) Proposing two lightweight implementations of our framework that are theoretically grounded. The algorithms are computationally and memory efficient, and achieve state-of-the-art results on CIFAR-10 and ImageNet when compared with competing methods. Furthermore, MANAS allows search directly on large datasets (e.g. ImageNet). (3) Presenting 3 news datasets for NAS evaluation to minimise algorithmic overfitting; offering a fair comparison with the often ignored random search (Li and Talwalkar, 2019) and random sampling (Yang et al., 2020; Yu et al., 2019) baselines; and presenting a complexity constraint analysis of MANAS.

2 Related work

MANAS derives its search space from DARTS (Liu et al., 2019) and is therefore most related to other gradient-based NAS methods that use the same search space. SNAS (Xie et al., 2019) appears similar at a high level, but has important differences: 1) it uses GD to learn the architecture parameters. This requires a differentiable objective (MANAS does not) and leads to 2) having to forward all operations (see their Eqs.5,6), thus negating any memory advantages (which MANAS has), and effectively requiring repeated cells and preventing search on ImageNet. Sequent gradient-based proposals improve upon baselines by introducing regularization mechanisms to improve the final performance of the generated architectures (Zela et al., 2018; Chen and Hsieh, 2020; Chu et al., 2021), whilst still suffering from the aforementioned problems.

ENAS (Pham et al., 2018) is also very different: its use of RL implies dependence on past states (the previous operations in the cell). It explores not only the stochastic reward function but also the relationship between states, which is where most of the complexity lies. Furthermore, RL has to balance exploration and exploitation by relying on sub-optimal heuristics, while MANAS, due to its theoretically optimal approach from online learning, is more sample efficient. Finally, ENAS uses a single LSTM (which adds complexity and problems such as exploding/vanishing gradients) to control the entire process, and is thus following a monolithic approach. Indeed, at a high level, our multi-agent framework can be seen as a way of decomposing the monolithic controller into a set of simpler, independent sub-policies. This provides a more scalable and memory efficient approach that leads to higher accuracy, as confirmed by our experiments.

Many proposals try to speed up NAS search. Zero-cost proxy estimators do this by evaluating architectures at initialisation (Lopes et al., 2021; Mellor et al., 2021; Chen et al., 2021; Abdelfattah et al., 2021). When coupled to memory-efficient NAS methods, such as random search, architecture search can be used in parallel and easily distributed to allow scoring several architectures simultaneously. GraphNAS++ leverages the idea of distributed evaluation by evaluating multiple architectures simultaneously in multiple GPUs to reduce the wall-clock time of a NAS search (Gao et al., 2022). Differently, AutoDistill partitions the search space and trains a supernetwork for each sub-partitioning, thus alleviating optimisation interference and speeding up the training process (Xu et al., 2022). Similarly, LaMOO (Zhao et al., 2021) and LaNAS (Wang et al., 2021b) progressively partition the search space based on regions with similar performance architectures. DC-NAS splits the search space based on the similarity of all sub-networks feature representations by clustering them using k-means (Wang et al., 2020). More closely related with MANAS are Federated NAS approaches (Zhu et al., 2021; Liu et al., 2022) in the sense of distributing the work through multiple agents. Federated NAS employs federated learning to allow individual users to optimise architecture weights and search for new architectures that match their data, as different clients usually have non-identical and independent data distributions (He et al., 2020; Yuan et al., 2020; Zhu and Jin, 2021; Garg et al., 2020; Hoang and Kingsford, 2021). However, neither distributing the search space by conducting parallel search on multiple GPUs nor independently distributing the search over different users mitigates the problem of large-scale optimisation of deep networks by itself. Since the required memory is still considerable, these approaches force NAS methods to search on a smaller number of cells, requiring heuristics to expand the searched cells into architectures. To mitigate these problems, MANAS leverages multi-agents, where each agent is associated with one layer selection and a global network is optimised based on each agent decision (detailed in Sect. 5). The multi-agent framework is inspired by a multi-arm bandit setting (Auer et al., 1995). Multi-arm bandit algorithms are a class of reinforcement learning algorithms used to balance exploration and exploitation in a decision-making process. This class of algorithms has been widely explored to solve a panoply of problems (Bouneffouf et al., 2020), such as designing recommendation systems (Li et al., 2010). By employing a multi-arm bandit, MANAS allows for a distributed space search through multiple agents with a global optimisation goal, and achieves both memory and time efficiency as a result. When compared with other NAS optimization methods that use the same search space, MANAS achieves better performance and requires less memory.

3 Preliminary: neural architecture search

We consider the NAS problem as formalised in DARTS (Liu et al., 2019). At a higher level, the architecture is composed of a computation cell that is a building block to be learned and stacked in the network. The cell is represented by a directed acyclic graph with V nodes and N edges; edges connect all nodes i, j from i to j where $i<j$. Each vertex $\varvec{x}^{(i)}$ is a latent representation for $i\in \{1,\ldots ,V\}$. Each directed edge (i, j) (with $i<j$) is associated with an operation $o^{(i,j)}$ that transforms $\varvec{x}^{(i)}$. Intermediate node values are computed based on all of its predecessors as $\varvec{x}^{(j)} = \sum _{i<j} o^{(i,j)}(\varvec{x}^{(i)})$. For each edge, an architect needs to intelligently select one operation $o^{(i,j)}$ from a finite set of K operations, ${\mathcal {O}} = \{o_{k}(\cdot )\}^K_{k=1}$, where operations represents some function to be applied to $\varvec{x}^{(i)}$ to compute $\varvec{x}^{(j)}$, e.g., convolutions or pooling layers. To each $o_{k}^{(i,j)}(\cdot )$ is associated a set of operational weights $w_{k}^{(i,j)}$ that needs to be learned (e.g. the weights of a convolution filter). Additionally, a parameter $\alpha _{k}^{(i,j)}\in {\mathbb {R}}$ characterises the importance of operation k within the pool ${\mathcal {O}}$ for edge (i, j). The sets of all the operational weights $\{w_{k}^{(i,j)}\}$ and architecture parameters (edge weights) $\{\alpha _{k}^{(i,j)}\}$ are denoted by $\varvec{w}$ and $\varvec{\alpha }$, respectively. DARTS defined the operation $\bar{o}^{(i,j)}(\varvec{x})$ as

$$\begin{aligned} \bar{o}^{(i,j)}(\varvec{x})= \sum _{k=1}^K \frac{e^{\alpha _{k}^{(i,j)}}}{\sum _{k'=1}^K e^{\alpha _{k'}^{(i,j)}}} \cdot o^{(i,j)}_k(\varvec{x}) \end{aligned}$$

(1)

in which $\varvec{\alpha }$ encodes the network architecture; and the optimal choice of architecture is defined by

$$\begin{aligned} \varvec{\alpha }^\star = \min _{\varvec{\alpha }} {\mathcal {L}}^{(\textrm{val})}( \varvec{\alpha },\varvec{w}^\star (\varvec{\alpha })) \quad \text {s.t.} \quad \varvec{w}^\star (\varvec{\alpha }) = \arg \min _{\varvec{w}} {\mathcal {L}}^{(\textrm{train})}(\varvec{\alpha },\varvec{w} ). \end{aligned}$$

(2)

The final objective is to obtain a sparse architecture $\mathcal {Z}^\star = \{\mathcal {Z}^{(i,j)}\}, \forall i,j$ where $\mathcal {Z}^{(i,j)}=[z_{1}^{(i,j)}, \dots ,z_{K}^{(i,j)}]$ with $z_{k}^{(i,j)}=1$ for k corresponding to the best operation and 0 otherwise. That is, for each pair (i, j) a single operation is selected.

Fig. 1

MANAS with single cell. Between each pair of nodes, an agent ${\mathcal {A}}_i$ selects action $a^{(i)}$ according to $\pi ^{(i)}$. Feedback from the validation loss is used to update the policy

4 Online multi-agent learning for AutoML

NAS suffers from a combinatorial explosion in its search space. A recently proposed approach to tackle this problem is to approximate the discrete optimisation variables (i.e., edges in our case) with continuous counterparts and then use gradient-based optimisation methods. DARTS (Liu et al., 2019) introduced this method for NAS, though it suffers from two important drawbacks. First, the algorithm is memory and computationally intensive (${\mathcal {O}}(NK)$ with K being total number of operations between a pair of nodes and N the number of nodes) as it requires loading all operation parameters into GPU memory. As a result, DARTS only optimises over a small subset of 8 repeating cells, which are then stacked together to form a deep network of 20. Naturally, such an approximation is bound to be sub-optimal. Second, evaluating an architecture on validation requires the optimal set of network parameters. Learning these, unfortunately, is highly demanding since for an architecture ${\mathcal {Z}}_{t}$, one would like to compute ${\mathcal {L}}^{(\textrm{val})}_{t}\left( \mathcal {Z}_t,\varvec{w}^{\star }_{t}\right)$ where $\varvec{w}^{\star }_{t} = \arg \min _{\varvec{w}} {\mathcal {L}}_{t}^{(\textrm{train})}(\varvec{w}, {\mathcal {Z}}_{t})$. DARTS, uses weight sharing that updates $\varvec{w}_t$ once per architecture, with the hope of tracking $\varvec{w}^{\star }_t$ over learning rounds. Although this technique leads to significant speed up in computation, it is not clear how this approximation affects the validation loss function.

Next, we detail a novel methodology based on a combination of multi-agent and online learning to tackle the above two problems (Fig. 1). Multi-agent learning scales our algorithm, reducing memory consumption by an order of magnitude from ${\mathcal {O}}(NK)$ to ${\mathcal {O}}(N)$; and online learning enables rigorous understanding of the effect of tracking $\varvec{w}^{\star }_{t}$ over rounds.

4.1 NAS as a multi-agent problem

To address the computational complexity we use the weight sharing technique used in DARTS. However, we handle in a more theoretically grounded way the effect of approximating ${\mathcal {L}}^{(\textrm{val})}_{t}\left( \mathcal {Z}_t, \varvec{w}^{\star }_{t}\right)$ by ${\mathcal {L}}^{(\textrm{val})}_{t}\left( \mathcal {Z}_t, \varvec{w}_{t}\right)$. Indeed, such an approximation can lead to arbitrarily bad solutions due to the uncontrollable weight component. To analyse the learning problem with no stochastic assumptions on the process generating $\nu = \{{\mathcal {L}}_{1}, \dots , {\mathcal {L}}_{T}\}$ we adopt an adversarial online learning framework.

NAS as Multi-Agent Combinatorial Online Learning. In Sect. 3, we defined a NAS problem where one out of K operations needs to be recommended for each pair of nodes (i, j) in a DAG. In this section, we associate each pair of nodes with an agent in charge of exploring and quantifying the quality of these K operations, to ultimately recommend one. The only feedback for each agent is the loss that is associated with a global architecture $\mathcal {Z}$, which depends on all agents’ choices.

We introduce N decision makers, ${\mathcal {A}}_{1}, \dots , {\mathcal {A}}_{N}$ (see Fig. 1 and Algorithm 1). At training round t, each agent chooses an operation (e.g., convolution or pooling filter) according to its local action-distribution (or policy) $\varvec{a}_{t}^{j} \sim \pi _{t}^{j}$, for all $j \in \{1, \dots , N\}$ with $\varvec{a}_{t}^{j}\in \{1, \dots , K\}$. These operations have corresponding operational weights $\varvec{w}_t$ that are learned in parallel. Altogether, these choices $\varvec{a}_{t}=\varvec{a}_{t}^{1}, \dots , \varvec{a}_{t}^{N}$ define a sparse graph/architecture $\mathcal {Z}_t\equiv \varvec{a}_{t}$ for which a validation loss ${\mathcal {L}}^{(\textrm{val})}_{t}\left( \mathcal {Z}_t,\varvec{w}_t\right)$ is computed and used by the agents to update their policies. After T rounds, an architecture is recommended by sampling $\varvec{a}_{T+1}^{j} \sim \pi _{T+1}^{j}$, for all $j \in \{1, \dots , N\}$. These dynamics resemble bandit algorithms where the actions for an agent ${\mathcal {A}}_{j}$ are viewed as separate arms. Multi-arm bandit algorithms allow balancing exploration and exploitation in a decision-making process. By employing a multi-arm bandit, MANAS allows for a distributed space search through multiple agents with a global optimisation goal, and achieves both memory and time efficiency as a result. This framework leaves open the design of 1) the sampling strategy $\pi ^{j}$ and 2) how $\pi ^{j}$ is updated from the observed loss.

Minimization of worst-case regret under any loss. The following two notions of regret motivate our proposed NAS method. Given a policy $\pi$ the cumulative regret ${\mathcal {R}}_{T,\pi }^{\star }$ and the simple regret $r_{T,\pi }^{\star }$ after T rounds and under the worst possible environment $\nu$, are:

$$\begin{aligned} {\mathcal {R}}_{T,\pi }^{\star }&= \sup _{\nu } {\mathbb {E}} \sum _{t=1}^{T} {\mathcal {L}}_{t}(\varvec{a}_{t}) - \min _{\varvec{a}} \sum _{t=1}^{T}{\mathcal {L}}_{t}(\varvec{a}), \end{aligned}$$

(3)

$$\begin{aligned} r_{T,\pi }^{\star }&= \sup _{\nu } {\mathbb {E}}\sum _{t=1}^{T} {\mathcal {L}}_{t}(\varvec{a}_{T+1}) - \min _{\varvec{a}}\sum _{t=1}^{T} {\mathcal {L}}_{t}(\varvec{a}) \end{aligned}$$

(4)

where the expectation is taken over both the losses and policy distributions, and $\varvec{a}=\{\varvec{a}^{({\mathcal {A}}_j)}\}^{N}_{j=1}$ denotes a joint action profile. The simple regret leads to minimising the loss of the recommended architecture $a_{T+1}$, while minimising the cumulative regret adds the extra requirement of having to sample, at any time t, architectures with close-to-optimal losses. We discuss in the Appendix E how this requirement could improve in practice the tracking of $\varvec{w}_t^\star$ by $\varvec{w}_t$. We let ${\mathcal {L}}_{t}(\varvec{a}_{t})$ be potentially adversarially designed to account for the difference between $\varvec{w}_t^\star$ and $\varvec{w}_t$ and make no assumption on its convergence. Our models and solutions in Sect. 5 are designed to be robust to arbitrary ${\mathcal {L}}_{t}(\varvec{a}_{t})$.

Because of the discrete nature of the NAS problem, during search the loss can take on large values or alternate between large and small values arbitrarily. Gradient-descent methods perform best under smooth loss functions, which is not the case in NAS. The worst-case regret minimization is a theoretically-grounded objective which we make use of in order to provide guarantees on the convergence of the algorithm when no assumptions are made on the process generating the losses.

5 Adversarial implementations

In the following subsections we will describe our proposed approaches for NAS when considering adversarial losses. We present two algorithms, MANAS and MANAS-LS, that implement two different credit assignment techniques specifying the update rule in line 7 of Algorithm 1. The first approximates the validation loss as a linear combination of edge weights, while the second handles non-linear losses.

Note that adversarial in this context refers to the adversarial multi-arm bandit (Auer et al., 1995) framework: we model the fact that a weight-sharing supernetwork returns noisy rewards as having an adversary that explicitly tries to confuse the learner. Adversarial multi-arm bandit is the strongest generalization of the bandit problem, as it removes all assumptions on the distribution. Our MA formulation and algorithm explicitly account for this adversarial nature and provide a principled solution that is provably robust.

5.1 MANAS-LS

5.1.1 Linear decomposition of the loss

A simple credit assignment strategy is to approximate edge-importance (or edge-weight) by a vector $\varvec{\beta }_{s}\in {\mathbb {R}}^{KN}$ representing the importance of all K operations for each of the N agents. $\varvec{\beta }_{s}$ is an arbitrary, potentially adversarially-chosen vector and varies with time s to account for the fact that the operational weights $\varvec{w}_{s}$ are learned online and to avoid any restrictive assumption on their convergence. The relation between the observed loss ${\mathcal {L}}_{s}^{(\textrm{val})}$ and the architecture selected at each sampling stage s is modeled through a linear combination of the architecture’s edges (agents’ actions) as

$$\begin{aligned} {\mathcal {L}}_{s}^{(\textrm{val})} = \varvec{\beta }_{s}^{\textsf{T}} \varvec{Z}_{s} \end{aligned}$$

(5)

where $\varvec{Z}_{s}\in \{0,1\}^{KN}$ is a vectorised one-hot encoding of the architecture ${\mathcal {Z}}_{s}$ (active edges are 1, otherwise 0). After evaluating S architectures, at round t we estimate $\varvec{\beta }$ by solving the following via least-squares:

$$\begin{aligned} \text {Credit assignment}:\,\,\, \widetilde{\varvec{B}}_{t} = \text {arg}\min _{\varvec{\beta }} \sum _{s=1}^{S} \left( {\mathcal {L}}_{s}^{(\textrm{val})} - \varvec{\beta }^{\textsf{T}} \varvec{Z}_{s}\right) ^{2}. \end{aligned}$$

(6)

The solution gives an efficient way for agents to update their corresponding action-selection rules and leads to implicit coordination. Indeed, in Appendix C we demonstrate that the worst-case regret ${\mathcal {R}}^\star _{T}$ (3) can actually be decomposed into an agent-specific form ${\mathcal {R}}^{i}_T\left( \varvec{\pi }^{i},\nu ^{i}\right)$ defined in the appendix: ${\mathcal {R}}^\star _{T} =\sup _{\nu }{\mathcal {R}}_{T}(\varvec{\pi }, \nu ) \iff \sup _{\nu ^{i}}{\mathcal {R}}^{i}_T \left( \varvec{\pi }^{i}, \nu ^{i}\right) , \ \ \ i=1,\ldots ,N$. This decomposition allows us to significantly reduce the search space complexity by letting each agent ${\mathcal {A}}_i$ determine the best operation for the corresponding graph edge.

Zipf Sampling for $r_{T,\pi }^{\star }$. ${\mathcal {A}}_i$ samples an operation k proportionally to the inverse of its estimated rank $\widetilde{ \langle k \rangle }^{i}_t$, where $\widetilde{ \langle k \rangle }^{i}_t$ is computed by sorting the operations of agent ${\mathcal {A}}_i$ w.r.t $\tilde{\varvec{B}}^{i}_t[k]$, as

$$\begin{aligned} \text {Sampling policy}:\,\,\,\varvec{\pi }^{i}_{t+1}[k]= 1 \Big /\widetilde{ \langle k \rangle }^{i}_t\overline{\log } K \quad \text { where } \overline{\log } K = 1+1/2+\ldots +1/K. \end{aligned}$$

Zipf explores efficiently, is anytime, parameter free, minimises optimally the simple regret in multi-armed bandits when the losses are adversarially designed and adapts optimally to stationary losses (Abbasi-Yadkori et al., 2018).

We prove for this new algorithm an exponentially decreasing simple regret $r^\star _{T}= {\mathcal {O}}\left( e^{-T/H}\right)$, where H is a measure of the complexity for discriminating sub-optimal solutions as $H=N(\min _{j\ne k^\star _i,1\le i \le N\} }\varvec{B}^{i}_T[j]-\varvec{B}^{i}_T[k^\star _i] )$, where $k^\star _i= \min _{1\le j \le K }\varvec{B}^{i}_T[j])$ and $\varvec{B}^{i}_T[j]=\sum _{t=1}^T \varvec{\beta }^{({\mathcal {A}}_i)}_t[j]$. The proof in given in Appendix D.1.

5.2 MANAS

5.2.1 Coordinated descent for non-linear losses

In some cases the linear approximation may be crude. An alternative is to make no assumptions on the loss function and have each agent directly associate the quality of their actions with the loss ${\mathcal {L}}^{(\textrm{val})}_t(\varvec{a}_t)$. This results in all the agents performing a coordinated descent approach to the problem. Each agent updates for operation k its $\widetilde{\varvec{B}}^{i}_{t}[k]$ as

$$\begin{aligned} \text {Credit assignment}:\,\, \widetilde{\varvec{B}}^{i}_{t}[k] = \widetilde{\varvec{B}}^{i}_{t-1}[k]+ {\mathcal {L}}^{(\textrm{val})}_t \cdot \mathbbm {1}_{ \varvec{a}_{t}^{i} = k }/\varvec{\pi }^{i}_t[k]. \end{aligned}$$

(7)

Softmax Sampling for ${\mathcal {R}}_{T,\pi }^{\star }$. Following EXP3 (Auer et al., 2002), actions are sampled from a softmax distribution (with temperature $\eta$) w.r.t. $\tilde{ \varvec{B}}^{i}_t[k]$:

$$\begin{aligned} \text {Sampling policy}:\,\, \varvec{\pi }^{i}_{t+1}[k] = \exp \left( \eta \tilde{\varvec{B}}^{i}_t[k] \right) \Big / \sum _{j=1}^{K}\exp \left( \eta \tilde{\varvec{B}}^{i}_t[j] \right) . \end{aligned}$$

Using this sampling strategy, EXP3 (Auer et al., 2002) is run for each agent in parallel. If the regret of each agent is computed by considering the rest of the agent as fixed, then each agent has regret ${\mathcal {O}}\left( \sqrt{TK\log K}\right)$ which sums over agents to ${\mathcal {O}}\left( N\sqrt{TK\log K}\right)$. The proof is given in Appendix D.1

On credit assignment. Our MA formulation provides a gradient-free, credit assignment strategy. Gradient methods are more susceptible to bad initialisation and can get trapped in local minima more easily than our approach, which, not only explores more widely the search space, but makes this search optimally according to multi-armed bandit derived regret minimization. Concretely, MANAS can easily escape from local minima as the reward is scaled by the probability of selecting an action (Eq. 7). Thus, the algorithm has a higher chance of revising its estimate of the quality of a solution based on new evidence. This is important as one-shot methods (such as MANAS and DARTS) change the network—and thus the environment—throughout the search process. Put differently, MANAS’ optimal exploration-exploitation allows the algorithm to move away from ‘good’ solutions towards ‘very good’ solutions that do not live in the former’s proximity; in contrast, gradient methods will tend to stay in the vicinity of a ‘good’ discovered solution.

6 Experiments

We (1) compare MANAS against existing NAS methods on the well established CIFAR-10 dataset; (2) evaluate MANAS on ImageNet; (3) compare MANAS, DARTS, Random Sampling and Random Search with WS (Li and Talwalkar, 2019) on 3 new datasets (Sport-8, Caltech-101, MIT-67); and (4) evaluate MANAS with inference time as complexity constraint. Descriptions of the datasets and details of the search are provided in the Appendix. We report the performance of two algorithms, MANAS and MANAS-LS, as described in Sect. 5. Note that, with the exception of results marked as +AutoAugment, all experiments were run with the same final training protocol as DARTS (Liu et al., 2019), for fair comparison.

Search Spaces. We use the same CNN search space as Liu et al. (2019). Since MANAS is memory efficient, it can search for the final architecture without needing to stack a posteriori repeated cells; thus, all our cells are unique. For fair comparison, we use 20 cells on CIFAR-10 and 14 on ImageNet. Experiments on Sport-8, Caltech-101 and MIT-67 in Sect. 6.3 use both 8 and 14 cell networks.

Search Protocols. For datasets other than ImageNet, we use 500 epochs during the search phase for architectures with 20 cells, 400 epochs for 14 cells, and 50 epochs for 8 cells. All other hyperparameters are as in Liu et al. (2019). For ImageNet, we use 14 cells and 100 epochs during search. In our experiments on the three new datasets we rerun the DARTS code to optimise an 8 cell architecture; for 14 cells we simply stacked the best cells for the appropriate number of times.

Synthetic experiment. To illustrate the theoretical properties of MANAS we apply it to the Gaussian Squeeze task, a problem where agents must coordinate their actions in order to optimize a global objective function that depends on the actions of each agent (Yang et al., 2018; Colby et al., 2015). Specifically, N homogeneous agents determine their individual actions $a^{(j)}$ to jointly optimize the objective $G(x) = x e^{-\frac{(x-\mu )^{2}}{\sigma ^{2}}}$ where $x = \sum _{j=1}^{N} a^{(j)}$. This synthetic setup has the same characteristics of the multi-agent NAS problem, namely a group of agents implicitly coordinating their actions to achieve a global objective, and is therefore a good experiment to showcase the theoretical properties of the MANAS algorithm.

We confirm that (1) MANAS progresses steadily towards zero regret while the Random Search baseline struggles to move beyond the initial starting poin; (2) MANAS stays well within the theoretical cumulative regret bound (Fig. 2).

Fig. 2

Left: Regret for the Gaussian Squeeze Domain experiment with 100 agents, 10 actions, $\mu =1$, $\sigma =10$. Right: Theoretical bound for the MANAS cumulative regret ($2N\sqrt{TK\log K}$; see Appendix D.2) and the observed counterpart for the Gaussian Squeeze Domain experiment with 100 agents, 10 actions, $\mu =1$, $\sigma =10$

Table 1 Comparison with state-of-the-art image classifiers on CIFAR-10

6.1 Results on CIFAR-10

6.1.1 Evaluation

To evaluate our NAS algorithm, we follow DARTS’s protocol: we run MANAS 4 times with different random seeds and pick the best architecture based on its validation performance. We then randomly reinitialize the weights and retrain for 600 epochs. During search we use half of the training set as validation. To fairly compare with more recent methods, we also re-train the best searched architecture using AutoAugment and Extended Training (Cubuk et al., 2018).

6.1.2 Results

Both MANAS implementations perform well on this dataset (Table 1). Our algorithm is designed to perform comparably to Liu et al. (2019) but with an order of magnitude less memory. However, MANAS actually achieves higher accuracy. The reason for this is that DARTS is forced to search for an 8 cell architecture and subsequently stack the same cells 20 times; MANAS, on the other hand, can directly search on the final number of cells leading to better results. We also report our results when using only 8 cells: even though the network is much smaller, it still performs competitively with 1st-order 20-cell DARTS. This is explored in more depth in Sect. 6.3. In terms of memory usage with a batch size of 1, MANAS 8 cells required only 1GB of GPU memory, while DARTSv1 utilized more than 8.5GB and DARTSv2 required 9.6GB, making both versions of DARTS unpractical to work with datasets with larger image sizes.

Cai et al. (2019) is another method designed as an efficient alternative to DARTS; unfortunately the authors decided to a) use a different search space (PyramidNet backbone; Han et al. (2017)) and b) offer no comparison to random sampling in the given search space. For these reasons we feel a numerical comparison to be unfair. Furthermore our algorithm uses half the GPU memory (they sample 2 paths at a time) and does not require the reward to be differentiable. Lastly, we observe similar gains when training the best MANAS/MANAS-LS architectures with an extended protocol (AutoAugment + 1500 Epochs + 50 Channels, in addition to the DARTS protocol).

6.2 Results on ImageNet

Table 2 Comparison with state-of-the-art image classifiers on ImageNet (mobile setting)

6.2.1 Evaluation

To evaluate the results on ImageNet we train the final architecture for 250 epochs. We report the result of the best architecture out of 4, as chosen on the validation set for a fair comparison with competing methods. As search and augmentation are very expensive we use only MANAS and not MANAS-LS, as the former is computationally cheaper and performs slightly better on average.

6.2.2 Results

We provide results for networks searched both on CIFAR-10 and directly on ImageNet, which is made possible by the computational efficiency of MANAS (Table 2). When compared to SNAS, DARTS, GDAS and other methods, using the same search space, MANAS achieves state-of-the-art results both with architectures searched directly on ImageNet and also with architectures transferred from CIFAR-10. We observe similar improvements when training the best MANAS architecture with an extended training protocol (AutoAugment + 600 Epochs + 60 Channels, in addition to the DARTS protocol), resulting in a final test error of 25.26% when directly searching on ImageNet.

6.3 Results on new datasets: Sport-8, Caltech-101, MIT-67

6.3.1 Evaluation

The idea behind NAS is that of finding the optimal architecture, given any sets of data and labels. Limiting the evaluation of current methods to CIFAR-10 and ImageNet could potentially lead to algorithmic overfitting. Indeed, recent results suggest that the search space was engineered in a way that makes it very hard to find a a bad architecture (Li and Talwalkar, 2019; Yu et al., 2019; Yang et al., 2020; Wan et al., 2022). To mitigate this, we propose testing NAS algorithms on 3 datasets (composed of regular sized images) that were never before used in this setting, but have been historically used in the CV field: Sport-8, Caltech-101 and MIT-67, described briefly in the Appendix. For these set of experiments we run the algorithm 8 times and report mean and std. We perform this both for 8 and 14 cells; we do the same with DARTS (which, due to memory constraints can only search for 8 cells). As baselines, we consider random search and random sampling. For the latter we simply sample uniformly 8 architectures from the search space. To efficiently implement random search, we follow Li and Talwalkar (2019) and perform experiments on random search with WS. Each proposed architecture is trained from scratch for 600 epochs as in the previous section.

6.3.2 Results

MANAS manages to outperform the random baselines and significantly outperform DARTS, especially on 14 cells (Fig. 3): this clearly shows that the optimal cell architecture for 8 cells is not the optimal one for 14 cells.

Fig. 3

Comparing MANAS, random sampling, random search with WS (Li and Talwalkar, 2019) and DARTS (Liu et al., 2019) on 8 and 14 cells. Average results of 8 runs. Note that DARTS was only optimised for 8 cells due to memory constraints

Table 3 Results of MANAS with complexity constraints using different penalty ($\lambda$) values on CIFAR-10

6.4 Results with complexity constraint

6.4.1 Evaluation

To evaluate the results of MANAS in a complexity constraint setting, we added the inference time of the generated architectures as a complexity constraint to the training. For this, we update the training loss with the constraint of the inference time that the generated architecture takes to classify an image: ${\mathcal {L}}^{(\textrm{train})}_t(\varvec{a}_t) ={\mathcal {L}}^{(\textrm{train})}_t(\mathcal {Z}_t,\varvec{w}_t )+\lambda L_t(\mathcal {Z}_t,\varvec{w}_t )$, where $L_t$ is the inference time to classify one image, and $\lambda$ defines the importance given to $L_t$. Here, the $\lambda$ serves the purpose of varying the importance given to the inference time whilst searching. By increasing $\lambda$, the inference time constraint has a higher importance.

6.4.2 Results

We evaluate MANAS with different $\lambda$ values using a single GPU (Table 3), and observe that by increasing the importance given to the inference, MANAS consistently generates architectures with a lower inference time, with similar accuracies. This experiment shows that MANAS can be extended for searching with multiple objectives, by modifying the training loss.

7 Discussion

Random Baselines. Clearly, in specific settings, random sampling performs very competitively. On one hand, since the search space is very large (between $8^{112}$ and $8^{280}$ architectures exist in the DARTS experiments), finding the global optimum is practically impossible. Why is it then that the randomly sampled architectures are able to deliver nearly state-of-the-art results? Previous experiments (Yu et al., 2019; Li and Talwalkar, 2019) together with the results presented here seem to indicate that the available operations and meta-structure have been carefully chosen and, as a consequence, most architectures in this space generate very good results. This suggests that human effort has simply transitioned from finding a good architecture to finding a good search space—a problem that needs careful consideration in future work. Random search with WS (Li and Talwalkar, 2019), has also shown to perform competitively but it is clearly sub-optimal compared to our multi-agent framework.

On fair evaluation. It is worth stressing that we performed all comparisons using the same final training protocol. This is extremely relevant as there has been a recent trend to boost results simply by stacking more training tricks on to the evaluation protocol. As such, any improvement in the final accuracy is solely due to how the network was trained rather than the quality of the search method used or the architecture discovered (Yang et al., 2020).

Agent coordination, combinatorial explosion and approximate credit assignment. Our set-up introduces multiple agents in need of coordination. Centralised critics use explicit coordination and learn the value of coordinated actions across all agents (Rashid et al., 2018), but the complexity of the problem grows exponentially with the number of possible architectures $\mathcal {Z}$, which equals $K^N$. We argue instead for an implicit approach where coordination is achieved through a joint loss function depending on the actions of all agents. This approach is scalable as each agent searches its local action space—small and finite—for optimal action-selection rules. Both credit assignment methods proposed learn, for each operation k belonging to an agent ${\mathcal {A}}_{i}$, a quantity $\widetilde{\varvec{B}}^{i}_{t}[k]$ (similar to $\alpha$ in Sect. 3) that quantifies the contribution of the operation to the observed losses.

8 Conclusions

We presented MANAS, a theoretically grounded multi-agent online learning framework for NAS. We proposed two extremely lightweight implementations that, within the same search space, outperform state-of-the-art while reducing memory consumption by an order of magnitude compared to Liu et al. (2019). We provide vanishing regret proofs for our algorithms. Furthermore, we evaluate MANAS on 3 new datasets, empirically showing its effectiveness in a variety of settings. Finally, we confirm concerns raised in recent works (Yu et al., 2019; Li and Talwalkar, 2019; Yang et al., 2020) claiming that NAS algorithms often achieve minor gains over random architectures. We however demonstrate, that MANAS still produces competitive results with limited computational budgets.

Appendices

A Datasets

CIFAR-10. The CIFAR-10 dataset (Krizhevsky, 2009) is a dataset with 10 classes and consists of 50, 000 training images and 10, 000 test images of size $32{\times }32$. We use standard data pre-processing and augmentation techniques, i.e. subtracting the channel mean and dividing the channel standard deviation; centrally padding the training images to $40{\times }40$ and randomly cropping them back to $32{\times }32$; and randomly flipping them horizontally.

ImageNet. The ImageNet dataset (Deng et al., 2009) is a dataset with 1000 classes and consists of 1, 281, 167 training images and 50, 000 test images of different sizes. We use standard data pre-processing and augmentation techniques, i.e. subtracting the channel mean and dividing the channel standard deviation, cropping the training images to random size and aspect ratio, resizing them to $224{\times }224$, and randomly changing their brightness, contrast, and saturation, while resizing test images to $256{\times }256$ and cropping them at the center.

Sport-8. This is an action recognition dataset containing 8 sport event categories and a total of 1579 images (Li and Fei-Fei, 2007). The tiny size of this dataset stresses the generalization capabilities of any NAS method applied to it.

Caltech-101. This dataset contains 101 categories, each with 40 to 800 images of size roughly $300{\times }200$ (Fei-Fei et al., 2007).

MIT-67. This is a dataset of 67 classes representing different indoor scenes and consists of 15, 620 images of different sizes (Quattoni and Torralba, 2009).

In experiments on Sport-8, Caltech-101 and MIT-67, we split each dataset into a training set containing $80\%$ of the data and a test set containing $20\%$ of the data. For each of them, we use the same data pre-processing techniques as for ImageNet.

B Implementation details

1.1 B.1 Methods

MANAS. Our code is based on a modified variant of Liu et al. (2019). To set the temperature and gamma, we used as starting estimates the values suggested by Bubeck et al. (2012): $t=\frac{1}{\eta }$ with $\eta =0.95\frac{\sqrt{\ln (K)}}{nK}$ (K number of actions, n number of architectures seen in the whole training); and $\gamma = 1.05 \frac{K\ln (K)}{n}$. We then tuned them to increase validation accuracy during the search.

MANAS-LS. For our Least-Squares solution, we alternate between one epoch of training (in which all $\beta$ are frozen and the $\omega$ are updated) and one or more epochs in which we build the Z matrix from Sect. 4 (in which both $\beta$ and $\omega$ are frozen). The exact number of iterations we perform in this latter step is dependant on the size of both the dataset and the searched architecture: our goal is simply to have a number of rows greater than the number of columns for Z. We then solve $\widetilde{\varvec{B}}_{t} = \left( \varvec{Z}\varvec{Z}^{\textsf{T}} \right) ^{\dagger }\varvec{Z}\varvec{L},$ and repeat the whole procedure until the end of training. This method requires no additional meta-parameters.

Number of agents. In both MANAS variants, the number of agents is defined by the search space and thus is not tuned. Specifically, for the image datasets, there exists one agent for each pair of nodes, tasked with selecting the optimal operation. As there are 14 pairs in each cell, the total number of agents is $14 \times C$, with C being the number of cells (8, 14 or 20, depending on the experiment).

1.2 B.2 Computational resources

ImageNet experiments were performed on multi-GPU machines loaded with $8\times$ Nvidia Tesla V100 16GB GPUs (used in parallel). All other experiments were performed on single-GPU machines loaded with $1\times$ GeForce GTX 1080 8GB GPU.

C Factorizing the Regret

Factorizing the Regret: Let us firstly formulate the multi-agent combinatorial online learning in a more formal way. Recall, at each round, agent ${\mathcal {A}}_i$ samples an action from a fixed discrete collection $\{\varvec{a}^{({\mathcal {A}}_i)}_{j}\}^{K}_{j=1}$. Therefore, after each agent makes a choice of its action at round t, the resulting network architecture ${\mathcal {Z}}_t$ is described by joint action profile $\varvec{\textbf{a}}_t =\left[ \varvec{a}^{({\mathcal {A}}_1),[t]}_{j_1}, \ldots , \varvec{a}^{({\mathcal {A}}_N),[t]}_{j_{N}} \right]$ and thus, we will use ${\mathcal {Z}}_t$ and $\varvec{\textbf{a}}_t$ interchangeably. Due to the discrete nature of the joint action space, the validation loss vector at round t is given by $\varvec{\mathbf {{\mathcal {L}}}}^{(\textrm{val})}_{t} =\left( {\mathcal {L}}^{(\textrm{val})}_{t}\left( {\mathcal {Z}}^{(1)}_t \right) , \ldots , {\mathcal {L}}^{(\textrm{val})}_{t} \left( {\mathcal {Z}}^{(K^N)}_t\right) \right)$ and for the environment one can write $\nu = \left( \varvec{\mathbf {{\mathcal {L}}}}^{(\textrm{val})}_{1}, \ldots , \varvec{\mathbf {{\mathcal {L}}}}^{(\textrm{val})}_{T}\right)$. The interconnection between joint policy $\varvec{\pi }$ and an environment $\nu$ works in a sequential manner as follows: at round t, the architecture ${\mathcal {Z}}_t\sim \varvec{\pi }_t(\cdot |{\mathcal {Z}}_1,{\mathcal {L}}^{(\textrm{val})}_{1},\ldots , {\mathcal {Z}}_{t-1}, {\mathcal {L}}^{(\textrm{val})}_{t-1})$ is sampled and validation loss ${\mathcal {L}}^{(\textrm{val})}_{t} = {\mathcal {L}}^{(\textrm{val})}_{t}({\mathcal {Z}}_t)$ is observed.¹ As we mentioned previously, assuming linear contribution of each individual action to the validating loss, one goal is to find a policy $\varvec{\pi }$ that keeps the regret:

$$\begin{aligned} {\mathcal {R}}_{T}(\varvec{\pi }, \nu ) = {\mathbb {E}}\left[ \sum _{t=1}^{T} \varvec{\beta }^{\textsf{T}}_{t}\varvec{Z}_t -\min _{\varvec{Z}\in {\mathcal {F}}} \left[ \sum _{t=1}^{T}\varvec{\beta }^{\textsf{T}}_{t} \varvec{Z}\right] \right] \end{aligned}$$

(8)

small with respect to all possible forms of environment $\nu$. We reason here with the cumulative regret the reasoning applies as well to the simple regret. Here, $\varvec{\beta }_t\in {\mathbb {R}}^{KN}_{+}$ is a contribution vector of all actions and $\varvec{Z}_t$ is binary representation of architecture ${\mathcal {Z}}_t$ and ${\mathcal {F}}\subset [0,1]^{KN}$ is set of all feasible architectures.² In other words, the quality of the policy is defined with respect to worst-case regret:

$$\begin{aligned} {\mathcal {R}}^*_{T} = \sup _{\nu }{\mathcal {R}}_{T}(\varvec{\pi }, \nu ) \end{aligned}$$

(9)

Notice, that linear decomposition of the validation loss allows us to rewrite the total regret (8) as a sum of agent-specific regret expressions ${\mathcal {R}}^{({\mathcal {A}}_i)}_T \left( \varvec{\pi }^{({\mathcal {A}}_i)}, \nu ^{({\mathcal {A}}_i)}\right)$ for $i=1,\ldots , N$:

$$\begin{aligned} {\mathcal {R}}_{T}(\varvec{\pi }, \nu )&= {\mathbb {E}} \left[ \sum _{t=1}^{T}\left( \sum _{i=1}^{N} \varvec{\beta }^{({\mathcal {A}}_i),\textsf{T}}_{t} \varvec{Z}^{({\mathcal {A}}_i)}_t - \sum _{i=1}^N \min _{\varvec{Z}^{({\mathcal {A}}_{i})} \in {\mathcal {B}}^{(K)}_{||\cdot ||_0, 1}(\varvec{0})} \left[ \sum _{t=1}^{T}\varvec{\beta }^{({\mathcal {A}}_i), \textsf{T}}_{t}\varvec{Z}^{({\mathcal {A}}_i)} \right] \right) \right] \nonumber \\&=\sum _{i=1}^N{\mathbb {E}}\left[ \sum _{t=1}^T \varvec{\beta }^{({\mathcal {A}}_i),\textsf{T}}_{t} \varvec{Z}^{({\mathcal {A}}_i)}_t -\min _{\varvec{Z}^{({\mathcal {A}}_{i})} \in {\mathcal {B}}^{(K)}_{||\cdot ||_0, 1}(\varvec{0})} \left[ \sum _{t=1}^{T}\varvec{\beta }^{({\mathcal {A}}_i), \textsf{T}}_{t}\varvec{Z}^{({\mathcal {A}}_i)}\right] \right] \\&= \sum _{i=1}^N{\mathcal {R}}^{({\mathcal {A}}_i)}_T \left( \varvec{\pi }^{({\mathcal {A}}_i)},\nu ^{({\mathcal {A}}_i)}\right) \end{aligned}$$

where $\varvec{\beta }_t = \left[ \varvec{\beta }^{{\mathcal {A}}_1, \textsf{T}}_{t},\ldots , \varvec{\beta }^{{\mathcal {A}}_N, \textsf{T}}_{t}\right] ^{\textsf{T}}$ and $\varvec{Z}_{t} =\left[ \varvec{Z}^{({\mathcal {A}}_1),\textsf{T}}_t,\ldots , \varvec{Z}^{({\mathcal {A}}_N),\textsf{T}}_t\right] ^{\textsf{T}}$, $\varvec{Z} =\left[ \varvec{Z}^{({\mathcal {A}}_1), \textsf{T}},\ldots , \varvec{Z}^{({\mathcal {A}}_N), \textsf{T}}\right] ^{\textsf{T}}$ are decomposition of the corresponding vectors on agent-specific parts, joint policy $\varvec{\pi }(\cdot ) =\prod _{i=1}^{N} \varvec{\pi }^{({\mathcal {A}}_i)}(\cdot )$, and joint environment $\nu = \prod _{i=1}^N\nu ^{({\mathcal {A}}_i)}$, and ${\mathcal {B}}^{(K)}_{||\cdot ||_0, 1}(\varvec{0})$ is unit ball with respect to $||\cdot ||_0$ norm centered at $\varvec{0}$ in $[0,1]^K$. Moreover, the worst-case regret (9) also can be decomposed into agent-specific form:

$$\begin{aligned} {\mathcal {R}}^\star _{T} = \sup _{\nu }{\mathcal {R}}_{T}(\varvec{\pi }, \nu ) \ \ \ \iff \ \ \ \sup _{\nu ^{({\mathcal {A}}_i)}} {\mathcal {R}}^{({\mathcal {A}}_i)}_T \left( \varvec{\pi }^{({\mathcal {A}}_i)}, \nu ^{({\mathcal {A}}_i)}\right) , \ \ \ i=1,\ldots ,N. \end{aligned}$$

This decomposition allows us to significantly reduce the search space and apply the two following algorithms for each agent ${\mathcal {A}}_i$ in a completely parallel fashion.

D Theoretical Guarantees

1.1 D.1 MANAS-LS

First, we need to be more specific on the way to obtain the estimates $\tilde{\varvec{\beta }}^{({\mathcal {A}}_i)}_t[k]$.

In order to obtain theoretical guaranties we considered the least-square estimates as in Cesa-Bianchi and Lugosi (2012) as

$$\begin{aligned} \tilde{\varvec{\beta }}_t={\mathcal {L}}^{(\textrm{val})}_t \varvec{P}^{\dagger }\varvec{Z}_t \text { where } \varvec{P} = {\mathbb {E}} \left[ \varvec{Z} \varvec{Z}^T \right] \text { with } \varvec{Z} \text { has law } \varvec{\pi }_t(\cdot ) = \prod _{i=1}^{N}\varvec{\pi }_t^{({\mathcal {A}}_i)}(\cdot ) \end{aligned}$$

(10)

Our analysis is under the assumption that each $\varvec{\beta }_t\in {\mathbb {R}}^{KN}$ belongs to the linear space spanned by the space of sparse architecture $\varvec{\mathcal {Z}}$. This is not a strong assumption as the only condition on a sparse architecture comes with the sole restriction that one operation for each agent is active.

Theorem 1

Let us consider neural architecture search problem in a multi-agent combinatorial online learning form with N agents such that each agent has K actions. Then after T rounds, MANAS-LS achieves joint policy $\{\varvec{\pi }_t\}^{T}_{t=1}$ with expected simple regret (Eq. 3) bounded by ${\mathcal {O}}\left( e^{-T/H}\right)$ in any adversarial environment with complexity bounded by $H=N(\min _{j\ne k^\star _i,i\in \{1, \ldots ,N\} }\varvec{B}^{({\mathcal {A}}_i)}_T[j] -\varvec{B}^{({\mathcal {A}}_i)}_T[k^\star _i] )$, where $k^\star _i= \min _{j\in \{1,\ldots ,K\}} \varvec{B}^{({\mathcal {A}}_i)}_T[j]$.

Proof

In Eq. 10 we use the same constructions of estimates $\tilde{\varvec{\beta }}_t$ as in ComBand. Using Corollary 14 in Cesa-Bianchi and Lugosi (2012) we then have that $\widetilde{\varvec{B}}_t$ is an unbiased estimates of $\varvec{B}_t$.

Given the adversary losses, the random variables $\tilde{\varvec{\beta }}_{t}$ can be dependent of each other and $t\in [T]$ as $\pi _{t}$ depends on previous observations at previous rounds. Therefore, we use the Azuma inequality for martingale differences by Freedman (1975).

Without loss of generality we assume that the loss ${\mathcal {L}}^{(\textrm{val})}_t$ are bounded such that ${\mathcal {L}}^{(\textrm{val})}_t\in [0,1]$ for all t. Therefore we can bound the simple regret of each agent by the probability of misidentifying of the best operation $P(k^\star _i\ne a^{{\mathcal {A}}_i}_{T+1})$.

We consider a fixed adversary of complexity bounded by H. For simplicity, and without loss of generality, we order the operations from such that $\varvec{B}^{({\mathcal {A}}_i)}_T[1] <\varvec{B}^{({\mathcal {A}}_i)}_T[2]\le \ldots \le \varvec{B}^{({\mathcal {A}}_i)}_T[K]$ for all agents.

We denote for $k>1$, $\Delta _{k} =\varvec{B}^{({\mathcal {A}}_i)}_T[k] -\varvec{B}^{({\mathcal {A}}_i)}_T[k^\star _i]$ and $\Delta _{1} =\Delta _{2}$.

We also have $\lambda _{min}$ as the smallest nonzero eigenvalue of $\varvec{M}$ where $\varvec{M}$ is $\varvec{M}=E[\varvec{Z} \varvec{Z}^T]$ where $\varvec{Z}$ is a random vector representing a sparse architecture distributed according to the uniform distribution.

$$\begin{aligned} P(k^\star _i\ne a^{{\mathcal {A}}_i}_{T+1})&= P\left( \exists k\in \{1,\ldots ,K\}: \widetilde{\varvec{B}}^{({\mathcal {A}}_i)}_T[1] \ge \widetilde{\varvec{B}}^{({\mathcal {A}}_i)}_T[k]\right) \\&\le P\left( \exists k\in \{1,\ldots ,K\}:\varvec{B}^{({ \mathcal {A}}_i)}_T[k]-\widetilde{\varvec{B}}^{({\mathcal {A}}_i)}_T[k]\right. \\&\left. \ge \frac{T\Delta _{k}}{2} \ \ \textrm{or}\ \ \widetilde{\varvec{B}}^{({\mathcal {A}}_i)}_T[1] -\varvec{B}^{({\mathcal {A}}_i)}_T[1] \ge \frac{T\Delta _{1}}{2}\right) \\&\le P\left( \widetilde{\varvec{B}}^{({\mathcal {A}}_i)}_{ T}[1] -\varvec{B}_T^{({\mathcal {A}}_i)}[1] \ge \frac{T\Delta _{1}}{2} \right) \\&\quad +\sum _{k=2}^{K} P\left( \varvec{B}_T^{({ \mathcal {A}}_i)}[k]-\widetilde{\varvec{B}}^{({ \mathcal {A}}_i)}_{T}[k] \ge \frac{T\Delta _{k}}{2}\right) \\&{\mathop {\le }\limits ^{{\textbf {(a)}}}} \sum _{k=1}^{K} \exp \left( -\frac{(\Delta _{k})^2T}{2Nlog(K)/\lambda _{min}}\right) \\&\le K\exp \left( -\frac{(\Delta _{1})^2T}{2Nlog(K)/\lambda _{min}}\right) , \end{aligned}$$

where (a) is using Azuma’s inequality for martingales applied to the sum of the random variables with mean zero that are $\tilde{\varvec{\beta }}_{k,t}-\varvec{\beta }_{k,t}$ for which we have the following bounds on the range. The range of $\tilde{\varvec{\beta }}_{k,t}$ is

$[0, Nlog(K)/\lambda _{min}]$. Indeed our sampling policy is uniform with probability 1/log(K) therefore one can bound $\tilde{\varvec{\beta }}_{k,t}$ as in (Cesa-Bianchi and Lugosi 2012, Theorem 1) Therefore we have $|\tilde{\varvec{\beta }}_{k,t}-\varvec{\beta }_{k,t} | \le Nlog(K)/\lambda _{min}$.

We recover the result with a union bound on all agents. $\square$

1.2 D.2 MANAS

We consider a simplified notion of regret that is a regret per agent where each agent is considering the rest of the agents as part of the adversarial environment. Let us fix our new objective as to minimise

$$\begin{aligned} \sum _{i=1}^N{\mathcal {R}}^{\star ,i}_{T}(\pi ^{({\mathcal {A}}_i)}) = \sum _{i=1}^N \sup _{\varvec{a}_{-i},\nu } {\mathbb {E}} \left[ \sum _{t=1}^{T} {\mathcal {L}}^{(\textrm{val})}_{t} (\varvec{a}^{({\mathcal {A}}_i)}_{t},\varvec{a}_{-i}) -\min _{\varvec{a}\in \{1,\ldots ,K\}} \left[ \sum _{t=1}^{T}{\mathcal {L}}^{(\textrm{val})}_{t} (\varvec{a},\varvec{a}_{-i})\right] \right] , \end{aligned}$$

where $\varvec{a}_{-i}$ is a fixed set of actions played by all agents to the exception of agent ${\mathcal {A}}_i$ for the T rounds of the game and $\nu$ contains all the losses as $\nu =\{{\mathcal {L}}^{(\textrm{val})}_{t}(\varvec{a})\}_{t\in \{1,\ldots ,T\},\varvec{a}\in \{1,\ldots ,K^N\} }$.

We then can prove the following bound for that new notion of regret.

Theorem 2

Let us consider the neural architecture search problem in a multi-agent combinatorial online learning form with N agents such that each agent has K actions. Then after T rounds, MANAS achieves joint policy $\{\varvec{\pi }_t\}^{T}_{t=1}$ with expected cumulative regret bounded by ${\mathcal {O}}\left( N\sqrt{TK\log K}\right)$.

Proof

First we look at the problem for each given agent ${\mathcal {A}}_i$ and we define and look at

$$\begin{aligned} {\mathcal {R}}^{\star ,i}_{T}(\pi ^{({\mathcal {A}}_i)}, \varvec{a}_{-i}) = \sup _{\nu } {\mathbb {E}}\left[ \sum _{t=1}^{T} {\mathcal {L}}^{(\textrm{val})}_{t}(\varvec{a}^{({\mathcal {A}}_i)}_{t}, \varvec{a}_{-i}) - \min _{\varvec{a}\in \{1,\ldots ,K\}} \left[ \sum _{t=1}^{T}{\mathcal {L}}^{(\textrm{val})}_{t} (\varvec{a},\varvec{a}_{-i})\right] \right] , \end{aligned}$$

We want to relate that the game that agent i plays against an adversary when the actions of all the other agents are fixed to $\varvec{a}_{-i}$ to the vanilla EXP3 setting. To be more precise on why this is the EXP3 setting, first we have that ${\mathcal {L}}^{(\textrm{val})}_{t}(\varvec{a}_t)$ is a function of $\varvec{a}_t$ that can take $K^N$ arbitrary values. When we fix $\varvec{a}_{-i}$, ${\mathcal {L}}^{(\textrm{val})}_{t}(\varvec{a}^{({\mathcal {A}}_i)}_{t}, \varvec{a}_{-i})$ is a function of $\varvec{a}^{({\mathcal {A}}_i)}_{t}$ that can only take K arbitrary values.

One can redefine ${\mathcal {L}}^{@,(\textrm{val})}_{t} (\varvec{a}^{({\mathcal {A}}_i)}_{t})={\mathcal {L}}^{(\textrm{val})}_{t} (\varvec{a}^{({\mathcal {A}}_i)}_{t},\varvec{a}_{-i})$ and then the game boils down to the vanilla adversarial multi-arm bandit where each time the learner plays $\varvec{a}^{({\mathcal {A}}_i)}_{t}\in \{1,\ldots ,K\}$ and observes/incurs the loss ${\mathcal {L}}^{@,(\textrm{val})}_{t}(\varvec{a}^{({\mathcal {A}}_i)}_{t})$. Said differently this defines a game where the new $\nu '$ contains all the losses as $\nu ' =\{{\mathcal {L}}^{@,(\textrm{val})}_{t} (\varvec{a}^{({\mathcal {A}}_i)}) \}_{t\in \{1,\ldots ,T\},\varvec{a}^{({\mathcal {A}}_i)}\in \{1,\ldots ,K\}}$.

For all $\varvec{a}_{-i}$

$$\begin{aligned} {\mathcal {R}}^{\star ,i}_{T}(EXP3, \varvec{a}_{-i}) \le 2\sqrt{TK\log (K)} \end{aligned}$$

Then we have

$$\begin{aligned} {\mathcal {R}}^{\star ,i}_{T}(EXP3)&\le \sup _{\varvec{a}_{-i}} 2 \sqrt{TK\log (K)}\\&= 2\sqrt{TK\log (K)} \end{aligned}$$

Then we have

$$\begin{aligned} \sum _{i=1}^N {\mathcal {R}}^{\star ,i}_{T}(EXP3)\le 2N\sqrt{TK\log (K)} \end{aligned}$$

$\square$

E Relation between weight sharing and cumulative regret

Ideally we would like to obtain for any given architecture $\mathcal {Z}$ the value ${\mathcal {L}}_{val}(\mathcal {Z},\varvec{w}^\star (\mathcal {Z}))$. However obtaining $\varvec{w}^\star (\mathcal {Z}) = \arg \min _{\varvec{w}} {\mathcal {L}}_{train}(\varvec{w}, \mathcal {Z})$ for any given fixed $\mathcal {Z}$ would already require heavy computations. In our approach the $\varvec{w}_t$ that we compute and update is actually common to all $\mathcal {Z}_t$ as $\varvec{w}_t$ replaces $\varvec{w}^\star (\mathcal {Z}_t)$. This is a simplification that leads to learning a weight $\varvec{w}_t$ that tend to minimise the loss ${\mathbb {E}}_{\mathcal {Z}\sim \pi _t}[{\mathcal {L}}_{val} (\mathcal {Z},\varvec{w}(\mathcal {Z})]$ instead of minimising ${\mathcal {L}}_{val}(\mathcal {Z}_t,\varvec{w}(\mathcal {Z}_t)$. If $\pi _t$ is concentrated on a fixed $\mathcal {Z}$ then these two previous expressions would be close. Moreover when $\pi _t$ is concentrated on $\mathcal {Z}$ then $\varvec{w}_t$ will approximate accurately $\varvec{w}^\star (\mathcal {Z})$ after a few steps. Note that this gives an argument for using sampling algorithm that minimise the cumulative regret as they naturally tend to play almost all the time one specific architecture. However there is a potential pitfall of converging to a local minimal solution as $\varvec{w}_t$ might not have learned well enough to compute accurately the loss of other and potentially better architectures.