Nat Comput

DOI 10.1007/s11047-016-9555-4

A Swarm Intelligence Based (SIB) method for optimization

Ó Springer Science+Business Media Dordrecht 2016

Abstract Natural heuristic methods, like the particle swarm called optimization. As scientific technology has advanced
optimization and many others, enjoy fast convergence and business operation has developed nowadays, opti-
towards optimal solution via inter-particle communications. mization problem exists in every corner of our living and
Many applications of such methods are applied to the opti- productivity, and it becomes a fundamental element of
mization in engineering, but only a few to the optimization in modern science. It has been widely applied to every sci-
statistics. It is especially difficult to implement in the opti- entific field, and thus, the research in optimization is
mization problems of experimental designs as the search always highly impact to most sciences. Boyd and Van-
space is mostly discrete, while most natural heuristic meth- denverghe (2004) provided a comprehensive study on how
ods are limited to searching continuous domains. This paper optimization is used in scientific fields and how it is
introduces a new natural heuristic method called Swarm important to today’s science and engineering. It has been a
Intelligence Based method for optimizing problem with a long history to the searches of the best optimization
discrete domain. It includes two new operations, MIX and approach. Linear programming is one of the most efficient
MOVE, for combining two particles and selecting the best optimization methods, but it is limited by the number of
particle respectively. This method is ready for the search of constraints, the linearity requirements and many others, so
both continuous and discrete domains, and its global best some variants like nonlinear prgramming, integer pro-
particle is guaranteed to monotonically move towards the gramming and dynamic programming appear to extend its
optimum. Several demonstrations on the optimization of use to a broader application. However, when a very large-
experimental designs are given at the end of this paper. scale problem is considered, these methods may not be the
most efficient choices.
Keywords Swarm intelligence  Designs of experiments  Inspired by the advances in biological technologies, one
Hadamard matrix  Supersaturated designs  Latin attempts to solve some complicated optimization problems
hypercube designs  Discrete domain from a biological point of view, and it leads to the devel-
opment of natural heuristics. It follows some natural
mechanisms by simulating the composition, evolution,
1 Introduction thinking, foraging and many other behaviors of human,
nature or animals, and constructs simulation models for
Nowadays there are many important problems that require searching the optimized solutions.
good solutions to improve productive efficiency under Furthermore, it is obvious in biology that the behavior of
some predefined conditions. These problems are generally a single subject is simple and its ability is limited. How-
ever, when a multiple of subjects work together, the out-
come from their cumulative abilities is more than a
& Frederick Kin Hing Phoa summation of every subject’s ability and the collective
fredphoa@stat.sinica.edu.tw
behavior is also much complicated. The corresponding
1
Institute of Statistical Science, Academia Sinica, Taipei 115, simulation model of optimization from these collective
Taiwan behaviors is generally considered as ‘‘group intelligence’’.

123
 F. K. H. Phoa

There were some common group intelligence methods in short compuational time, flexibile forms of optimum and
the literature, which includes Genetic Algorithm Fraser constraints, small number of tuning parameters and minimal
(1957), Memetic Algorithm Dawkins (1976), Particle memory space for mathematical operations.
Swarm Optimization (PSO) Kennedy and Eberhart (1995), Let f : RK ! R be a given known function for mini-
Shuffled Frog Leaping Algorithm Eusuff et al. (2006), Cat mization over a known set S, given S ¼ fx 2
Swarm Optimization Orouskhano et al. (2013), Bacterial RK jlj  xj  uj ; j ¼ 1; . . .; Kg In our design problem,
Foraging Optimization Passino (2002), Ant Colony Algo- f represents the design criterion or the objective function
rithm Dorigo (1992), Artificial Bee Colony Algorithm and S is the design space. The domain of constraints are not
Seeley (1995) and many others. limited to box constraints. It can be circular Panduro et al.
In this paper, we propose a new natural heuristic approach (2009), elliptical or concentric elliptical Bera and Roy
called ‘‘Swarm Based Intelligence’’ (SIB) algorithm. It is (2013), depending on the practitioners’ needs.
ready for optimization in either discrete or continuous The standard PSO algorithm begins by first randomly
domains, but we only consider discrete domains due to the generating a swarm of particles to search for the optimal
theme of this paper that optimizes experimental designs. solution. The size of the swarm is user-selected and each
Section 2 reviews the basic of PSO and points out the limits particle in the swarm represents a candidate solution to the
of PSO in searching discrete domains. Section 3 provides a optimization problem. At any particular time, each particle
general framework of the SIB algorithm, which includes has its own perceived location of the optimum in the high-
detailed description of the steps and operations in the SIB dimensional search space and with each iteration, it moves
algorithm. Section 4 demonstrates how the SIB approach is towards it and then updates its perceived optimum location.
used in searching circulant partial Hadamard matrices with This perceived optimum location by the particle at a given
maximum number of columns. Section 5 briefly describes iteration is often called the ‘‘Local Best’’ (LB). Throughout,
other existing applications of the SIB algorithm. Discussions the particles share information on the perceived overall
on the difference between SIB and PSO and how SIB is used optimum location; this position is called ‘‘Global Best’’
in continuous domain are given the last section, together with (GB). In each iteration, every particle is accelerated towards
a brief summary of this paper. its LB in the direction of the GB position with a velocity
based on a random weighing on its current distance from its
LB and its current distance from the GB position. For the ith
2 A review on particle swarm optimization particle after completing its nth iteration, the following two
equations show how the new location xi;nþ1 is determined
PSO is a population-space-based optimization tool pro- and what velocity vi;nþ1 to use to get there:
posed by Kennedy and Eberhart (1995). The framework
was followed by Shi and Eberhart (1998), Eberhart and Shi vi;nþ1 ¼ vi;n þ C1 ðxGB  xi;n Þ þ C2 ðxi;LB  xi;n Þ
2001) and many others. Prior to the current use in com- xi;nþ1 ¼ xi;n þ vi;nþ1  dt:
putational intelligence and optimization, it was originally
Here vi;j and xi;j are respectively the velocity and position
used for the study of social network structure. Clerc (2006),
of particle i at iteration j, C1 and C2 are the stochastic
Engelbrecht (2005) and other textbooks provided compre-
terms, xGB is the GB position as perceived by all particles
hensive introduction on the optimization theories, evolu-
and xi;LB is the LB position for particle i. The new location
tionary computations and practical applications of PSO and
for the next iteration is the sum of its current location and
its variants. In general, PSO is capable of solving many
the velocity it uses to get there at unit time when dt ¼ 1.
different types of high-dimensional optimization real world
The search continues until a user-defined stopping criterion
problems with multiple optima. Some applications of PSO
is met. Examples of stopping criteria are (i) when all par-
in computer science and engineering include training
ticles locate the GB position and the GB position does not
neural networks (Braendler and Hendtlass 2002), solving
change any more after a user-defined number of iterations,
dynamic economic dispatch problems Chakrabarti et al.
or (ii) a time limit is reached or (iii) a pre-specified max-
(2006), pole shape optimization problems (Brandstatter and
imum number of iterations jmax is reached.
Baumgartner 2002), quadratic assignment problems Gong
and (Tuson 2008) and many others.
PSO begins with a population of particles and the whole 3 A general framework of the Swarm Intelligence
population is traced throughout the search. The iterative Based (SIB) algorithm
procedure of PSO relies on heuristics and stochastics and it
has been shown in many applied disciplines that PSO usually It is commonly known that, contrary to many other algo-
converges or nearly converges to the global optimum. Phoa rithm, PSO does not require any assumption of the objec-
et al. (2016) commented that PSO is advantagous in terms of tive function to be optimized. For example, linear

123
A Swarm Intelligence Based (SIB) method for optimization in designs of experiments

programming requires the objective function be linear and large number of particle generally increases the
the widely used Newton-Rhapson’s method requires that computational time to the convengence, but it requires
the objective function be differentiable. However, accord- a large memory for mathematical operation.
ing to the general framework of PSO in the last section, it is 3. Numbers of discrete units being exchanged with LB
obvious that the search domain has to be continuous. and GB particles ( qLB and qGB ). They are optimal
In this section, we follow ideas from swarm intelligence parameters and two numbers can be different. A large
to propose an algorithm for solving discrete optimization value of these numbers generally increases the speed of
problem. This method, which is generally called Swarm particles to move towards the best particles, but it
Intelligence Based (SIB) method, was first proposed in increases the possibility that overlooks the potentially
Phoa et al. (2016) for design optimization problem. In good particles in between the current particle and the
specific, it suggested a SIBSSD algorithm for searching best particle. In other words, it increases the occurance
supersaturated designs (SSDs) with optimal Eðs2 Þ criterion. of random jumps and reduces the effects of particles’
A SSD is an experimental plan that is mainly designed for inter-communications.
screening purposes. It can be viewed as a rectangular Note that similar to PSO, at any particular time, each
matrix that the number of rows is much smaller than the particle has its own perceived location of the optimum in
number of columns. Statistically speaking, it is a ‘‘small-n- the search space and it is called the LB. Throughout, N
large-p’’ problem that the degree of freedoms are not particles share information by comparing its LB with other
enough for independently estimating all main effects LBs and collectively arrived at the perceived overall
without biases. This section proposes a general framework optimum location called the GB. In addition, there are no
of the SIB algorithm for a general discrete problem, and the theoretical results to decide the best values of qLB and qGB
applications in design of experiments are demonstrated in and they can only be decided roughly by experience.
next sections. However, it is wise to set qGB \qLB to avoid premature
Table 1 shows the main steps where each particle is first convergence to a relatively good particle without adequate
mixed with selected particles and then moved or replaced exploration of the search space..
by another particle by the MOVE operation discussed Once the parameters are entered, N particles are gener-
below. ated and the values of the objective function for these
particles are then evaluated. Since there are no prior
3.1 Initialization step objective function values to compare with at the initial
step, we default these particles to be the LB particles. The
The initialization step can be viewed as the zeroth iteration particle with the optimal value of the objective function
of the SIB algorithm. It receives input parameters from the among all particles is the GB particle. Following the ini-
users and returns the information of the local and global tialization step, the iteration starts from the while-loop in
best particles. Users are required to enter several initial Table 1. First, the SIB algorithm performs the MIX oper-
parameters and settings listed below: ation to update all particles by exchanging qLB or qGB
1. Stopping criterion It is the objective function for number of units with the corresponding best particle. Then
evaluation in the problem. in the MOVE operation, the SIB algorithm moves the
2. Number of particles (N) It is the number of randomly current particle closer to the best particle by selecting the
generated particles being used in the SIB algorithm. A best candidates among three generated in the MIX opera-
tion. Following similar fashion in Phoa et al. (2016), we
now describe the MIX and MOVE operations in details.
Table 1 The SIB algorithm
1: Randomly generate a set of initial particles
2: Evaluate objective function value of each particle 3.2 MIX operation
3: Initialize the LB for all particles
4: Initialize the GB
We consider a candidate particle X and another particle Y,
5: while not converge do
which is either the LB or the GB particles. The MIX
operation consists of deleting units from X and then adding
6: For each particle, perform the MIX operation
units from Y. The general idea is to improve the values of
7: For each particle, perform the MOVE operation
objective functions of X via substituting q ‘‘bad’’ units of X
8: Evaluate objective function value of each particle
by q ‘‘good’’ unit of Y. q represents qLB or qGB when Y is
9: Update the LB for all particles
the LB or the GB particles respectively.
10: Update the GB
Assume that there are m discrete units in every particle
11: end while
X. The deletion step for X is a recursive procedure that

123
 F. K. H. Phoa

determines which unit is the best unit to be removed from updated only if the best LB particle has a better value. After all
X. We first construct all m reduced particles X1 ; . . .; Xm , current particles, LB particles and the GB particle are upda-
where Xi consists of all units of X except the ith unit ted, the search continues to its next iteration in the same way
(1  i  m) and assume that among all these particles now until the pre-specified maximum number of iterations is
with m  1 units, Xd has the best objective function value. reached or the GB particle reaches its optimal value if known.
We delete unit d from X in the current step and treat the
reduced particle as X in the next step. This deletion pro-
cedure continues sequentially until q units are deleted from
the original X. We denote the reduced particle by R. 4 A demostration on circulant partial Hadamard
The addition step is the reverse of the deletion step, matrices
where q units from Y are sequentially added in a similar
way to R. We first create an expanded particle Ri ¼ ½R; yi , 4.1 A brief introduction to circulant partial
where Ri contains all units in R and yi is the ith unit from Y, Hadamard matrices
i ¼ 1; . . .; m. The selected expanded particle is Rd ¼ ½R; yd 
if the objective function value of Rd is the best among all A Hadamard matrix of order n is an n  n square matrix H
Ri . The process continues until q units are added to the with entries ð1Þ such that HH T ¼ nIn , which also sug-
reduced particle R. gests that its rows are orthogonal and thus designs based on
The MIX operation for X results in a new particle called Hadamard matrices are D-optimal. The studies of Hada-
mixwGB when Y is the GB particle and another new paricle mard matrices and their design properties are rich, see
called mixwLB when Y is the LB particle. These two Geramita and Seberry (1979), Horadam 2007) and others
resulting particles incorporate the component units of the for details. It is well-known that there are some connec-
LB and GB particles in the search for the new particle in tions between combinatorial and experimental designs. For
the next iteration while retaining partial component units example, the Plackett–Burman design (1946) is a famous
from the candidate particle. This completes the MIX Hadamard matrix of two levels. Such design has been
operation for X. widely used in the experiments with 12 runs and at most 11
factors. Additional details about the uses of Plackett–Bur-
3.3 MOVE operation man designs or other orthogonal arrays as fractional fac-
torial exierpiemtnal plans are referred to Xu et al. (2009).
The MOVE operation comes after the MIX operations of Low et al. (2005) pioneered a study of partial Hadamard
all X are completed and there are three candidates, matrix, which is a k  n rectangular matrix with entries ð1Þ
mixwGB, mixwLB and X, under consideration. The MOVE such that HH T ¼ nIk . Craigen et al. (2013) introduced a
operation is a decision-making procedure to select the best class of k  n partial Hadamard matrices that are constructed
particles among all cnadidates with the optimal value of by circulating any row of the matrix k times. Such row is
objective functions. The movement or update of X is then generally called a generating vector. They denote ‘‘r-
accomplished by replacing it with the selected best particle. Hðk  nÞ’’ as a k  n circulant partial Hadamard matrix
It is obvious that if either mixwGB or mixwLB has the best (CPHM) such that the numbers of þ1 and 1 in every row of
value evaluated from the objective function, we replace X H differ by r. It is a useful class of design and a good suc-
by the best candidate. However, if both mixwGB and cessor of traditional m-sequences for biometical experiments
mixwLB are worse than X, some units of X are randomly with functional magnetic resonance imaging (fMRI). Table 1
chosen and replaced by some other random units. We in Craigen et al. (2013) listed the maximum k for which r-
recommend that the number of randomly replaced units Hðk  nÞ exists, it was followed by several lemmas that
should not be larger than qLB , which is sufficient to escape describe the relationship among r, k and n. However, they do
from the trap of a local optimum and to explore unsearched not provide the exact generating vectors of 1 for each
domain. We denote the particle X updated in this way by H reported in Table 1. Not to mention some H of large order
mixwRC and such procedure is called the random jump. n, they could only provide possible ranges of maximum k.
After each particle has completed the MOVE operation, We formally define the circulant Hadamard matrix and
the objective function values of all particles are evaluated circulant partial Hadamard matrix here.
again and compared to the values of the individual LB par-
Definition 1 Consider a k  n matrix A ¼ faij g with first
ticles. If the updated particle has a better value than that of its
row ða1 ; a2 ; . . .; an Þ.
LB particle, the current particle is considered as the new LB
particle. Otherwise no change is made to the LB particle. (a) A is a circulant if each row after the first is obtained
Next, the value of the best LB particle among all LB particles by a right cyclic shift of its predecessor by one
is compared to that of the GB particle, and this design is position, thus aij ¼ aðjiÞmodn for i ¼ 0; . . .; k  1;

123
A Swarm Intelligence Based (SIB) method for optimization in designs of experiments

(b) A is a circulant partial Hadamard matrix if and only Table 2 SIBCPHM algorithm
if it is circulant and AAT ¼ nIk ; 1: Randomly generate N vectors of length n as initial vectors
(c) When n ¼ k, A is a circulant Hadamard matrix of 2: Evaluate the maximum value of k of each vector
order n. 3: Initialize the LB vectors for all vectors
4: Initialize the GB vector
Example 1 We consider a generating vector
5: while not converge do
a ¼ ð; ; ; þ; ; þ; þ; þÞ. Let A be a matrix and a be
6: For each vector, perform the MIX operation
its first row, which makes ðþ; ; ; ; þ; ; þ; þÞ and
7: For each vector, perform the MOVE operation
ðþ; þ; ; ; ; þ; ; þÞ its second and third rows
8: Evaluate the maximum value of k of each vector
respectively. A that consists of these three rows is a 0-
9: Update the LB vector for all vectors
Hð3  8Þ: (i) r ¼ 0 because the number of þ and - entries
10: Update the GB vector
in each row are equal; (ii) A has a dimension of 3 rows and
11: end while
8 columns; (iii) AAT is a 3  3 diagonal matrix with all
diagonal entries 8. In fact, according to Table 1 in Craigen
et al. (2013) and our search, the maximum possible k for A
with r ¼ 0 and n ¼ 8 is 3. optimal settings on exchanging rates from professional
opinion, users can manually overwrite these rates.
The circulant nature greatly reduces the search space of Following the initialization step, the first iteration starts
optimal CPHM from 2kn to 2n , because the structure of the from the while-loop in Table 2. The particle of the SIB
generating vector of length n governs the matrix properties algorithm is the generating vector in the SIBCPHM algo-
completely. This motivates us to search for an efficient rithm. The discrete unit to be exchanged refers to the þ and
algorithm that can be used to find these exact generating - entry in the generating vector, and the number of dis-
vectors, and hopefully this algorithm is efficient enough so crete units m is the length of the vector. During the MIX
that the length n will not be a serious factor in computa- operation, q entries of the candidate vector is deleted
tional burden. It leads to the use of natural heuristic sequentially for each vector. After that, q entries are filled
algorithm. in the candidate vector from vector Y, where Y is the LB
vector (q ¼ qLB ) or the GB vector (q ¼ qGB ), resulting in
4.2 The SIBCPHM algorithm two intermediate vectors mixwLB and mixwGB respec-
tively. Then the MOVE operation compares two interme-
It is obvious that the optimized target, i.e. the generating diate vectors and the candidate vector and selects the
vector, is not continuous and thus many natural heuristic vectors that lead to the higher maximum k value. Such
algorithms like PSO cannot be directly applicable for selection is viewed as an update.
finding optimal circulant partial Hadamard matrix. The In order to further enhance the efficiency during the
search space is a large set of entries comprising of 1 and exchange, the selection of entries that is used to be
the optimization problem is to select a right combination of exchanged is slightly modified as non-random. Prior to the
n entries for the generating vector. To achieve this goal, we selection, there is a detection on the difference between the
apply the SIB algorithm and redefine the update operation candidate vector and the LB or GB vector. These locations
as Table 2, and name the procedure SIBCPHM (Swarm of different entries are then ranked according to the impact
Intelligence Based Circulant Partial Hadamard Matrix) on the value of k. The first qLB or qGB entries with the
algorithm. highest value of k are exchanged. Such procedure ensures
Prior to the start of SIBCPHM, users are required to that the exchange leads to the best possible result. Similar
enter some parameters, which include matrix-related to the SIB algorithm, if the two intermediate vectors are not
parameters (r; n) and SIB-related parameters (N, number of better than the original vector, the MOVE operation will
seeds; t, number of maximum iteration; qLB , LB exchange generate a new vector to restart. In other words, the restart
rate; qGB , GB exchange rate). Steps 1 to 4 are the initial- not only allows the vector to jump out from the local
ization steps of SIBCPHM. It starts with N vectors of attractor, but also to explore more search space.
length n that are created randomly, and the number of þ Following the MOVE operation in each iteration, the
and - in each vector differ by r. Users also need to input t maximum values of k of each vector are evaluated again
as a stopping criterion. It is not mandatory for the users to and are compared to those of their individual LB vectors.
input two exchange rates, because qLB ¼ bn=3c and qGB ¼ For some vectors with better maximum k values, their
bqLB =2c are set as default. Although they are not theoret- individual LB vectors are updated. Then the best LB vector
ically derived, such settings work quite well in our expe- among all is compared to the GB vector and it is updated
rience. However, if there are some prior information of when the best LB vector has a higher maximum k value.

123
 F. K. H. Phoa

The SIBCPHM algorithm terminates in two conditions. vectors as binary codes, which is a common abbreviation
The first condition is when the maximum value of k attains method in the field of experimental designs.
its best possible value. Some of these upper bounds can be This table provides the results only from n ¼ 8 to n ¼
found in Table 1 of Craigen et al. (2013). The second 32 because the vectors become too long for larger n. In
condition is when the number of iteration reaches t. Once fact, we have successfully obtained the generating vectors
SIBCPHM is stopped, the GB is the generating vector up to n ¼ 52 for r ¼ 0 and n ¼ 76 for r ¼ 2. For r ¼ 0, our
suggested via SIBCPHM. result is the largest definitive result reported in Craigen
et al. (2013); however, this result came up only after a few
hours’ search using a personal computer with Intel Core i7-
4.3 Circulant partial Hadamard matrices 3770 CPU and 8GB RAM. For r ¼ 2, our result has gone
with maximum number of columns much further than what Craigen et al. (2013) has reported
in their Table 1. Results for large n are available upon
The search results of using SIBCPHM are listed in Table 3. request from the first author.
The first two columns are the length of the generating Moreover, we only report the cases of r ¼ 0 and r ¼ 2
vector (or column number of the matrix) and the difference in this paper, but there are other settings r [ 2 shown in
between the numbers of þ and -. The third column is the Craigen et al. (2013) that we do not report here. First,
maximum k values obtained from Table 1 of Craigen et al. notice that Craigen et al. (2013) has derived some theo-
(2013), and all the listed results achieved their upper retical results from very large r but we are not interested in
bounds. repeating the known results. For the case of r ¼ 0, this
The last column is the resulting generating vectors. class is considered as balanced designs in the field of
Notice that these vectors are being cycled so that the experimental designs and thus it is highly compelling. For
longest consecutive—is located in the front. For example, the case of r ¼ 2, this class possesses maximum k among
for the first case n ¼ 8 and r ¼ 0, SIBCPHM may return a all different r values. In other words, this class of matrix is
generating vector ðþ þ þ    þÞ and it is trivial that able to generate experimental designs with maximum
this result is equivalent to the one in the table after three- number of two-level factors which are uncorrelated when
entry left-shifts. Furthermore, it is also clear that once a experiments are analyzed. It is true that there are also some
generating vector is obtained, any vectors resulted from r ¼ 4 cases with equivalently maximum k, such as n ¼ 12
shifts are still generating vectors for the circulant partial [generating vector: ð    þ    þ  þþÞ] and
Hadamard matrix of the same size. We list these generating n ¼ 28 [generating vector: ð     þ  þ þ  þ þ
vectors with the longest consecutive - at the front in  þ þ þ  þ þ þ þ  þ  þ þþÞ]. Although we do
Table 3, because it looks neat. What is more, it will return not report the results in the table, users can still use
the smallest values of decimal counterpart if we treat these SIBCPHM to search for their desired cases of r.

Table 3 Results from

8 0 3 ð þ  þ þþÞ
8 2 4 ð þ þ  þÞ
12 0 5 ð þ þ  þ þ þ  þÞ
12 2 6 ð þ  þ þ þ þÞ
16 0 7 ð þ  þ  þ þ þ  þ þþÞ
16 2 8 ð  þ þ þ þ  þ  þ þÞ
20 0 7 ð þ  þ  þ þ þ þ þ þ  þþÞ
20 2 10 ð þ  þ  þ þ þ þ þ þ  þÞ
24 0 9 ð þ  þ  þ þ þ þ  þ  þ þ þ  þþÞ
24 2 12 ð þ þ þ  þ þ  þ  þ þ þ þ þÞ
28 0 9 ð þ  þ  þ þ þ þ þ þ þ  þ þ  þ  þþÞ
28 2 14 ð þ  þ  þ þ  þ þ þ þ  þ  þ þ þ þÞ
32 0 12 ð þ  þ  þ þ  þ þ þ  þ þ þ þ  þ þ  þ þ þÞ
32 2 14 ð þ þ þ  þ þ þ  þ  þ þ þ þ þ  þ  þ þÞ

123
A Swarm Intelligence Based (SIB) method for optimization in designs of experiments

5 Other applications two mixed SSDs, a total of qLB columns indexes are ran-
domly chosen from a pool consisting of all balanced col-
5.1 Eðs2 Þ-Optimal supersaturated designs umns of length N and they are randomly substituted to the
candidate SSD. The resulting SSD, namely mixwRC, is
An application of the SIB algorithm to the optimization of considered as an update.
Eðs2 Þ-optimal supersaturated designs (SSDs) was proposed The SIBSSD algorithm successfully searches for bal-
in Phoa et al. (2016). A two-level SSD with N runs and m anced SSDs with high Eðs2 Þ values, if not Eðs2 Þ-optimal,
factors is represented by an N  m matrix X with every for a list of N and m comparable to Bulutoglu and Cheng
entry equals to þ or -. It is also assumed that N\m for (2004) and Bulutoglu (2007). Readers who are interested in
run size economy and no two columns of X are completely detailed results can refer to Phoa et al. (2016).
identical to one another. A SSD is balanced if þ and -
levels of each factor appears in the design the same number 5.2 /P -Optimal Latin hypercube designs
of times. The balance property implies orthogonality
among main effects and interactions, so each factor is The optimization of /P -optimal Latin hypercube designs
estimated and tested as if it were the only one under con- (LHDs), which was proposed in Chen et al. (2013), can be
sideration, with very little loss of efficiency due to the recast as an application of the SIB algorithm. An n-run and
presence of other factors. Box and Cox (1962) proposed a k-factor LHD is an n by k matrix. Each row of the matrix
formal approach by minimizing the average nonorthogo- corresponds to a design point and each column is a per-
nality between all pairs of columns of the model matrix as mutation of f1; 2; . . .; ng. It has the one dimensional pro-
a design criterion called Eðs2 Þ criterion defined as follow jection property, meaning that by projecting an n-point
X m design onto any factor, there are exactly n different levels
Eðs2 Þ ¼ s2ij = for that factor. Although a LHD is guaranteed to be non-
i\j
2
collapsing, a randomly generated LHD may not always be
where sij is the dot product between the ith and jth columns spacefilling. To rank the competing LHDs achieving the
of X. space-filling property, Morris and Mitchell (1995) pro-
Phoa et al. (2016) proposed a SIB algorithm, namely posed the /P criterion
SIBSSD, for searching N  m SSDs with optimal Eðs2 Þ !1=p
X m
p
criterion. The initialization steps start with randomly cre- /P ¼ J k dk
ated N SSDs that each SSD is represented by a vector of k¼1
m column indexes. A column index is a common method in where p is a positive integer, d1 ; . . .; dm are the numbers of
experimental designs to represent a two-level column such sorted distinct inter-site distances from minimum to max-
that the column is the binary number of a decimal column imum and Jk is the number of inter-site distances that are
index. These column indexes are the discrete units to be equal to dk .
exchanged in the SIBSSD algorithm. Users enter the Chen et al. (2013) proposed a LaPSO algorithm for
standard inputs including exchange rates qLB , qGB and searching n  k /P -optimal LHDs. The initialization steps
maximum number of iterations t in the initialization steps. start with randomly creating N LHDs that each LHD is
The MIX operation in the SIBSSD algorithm is a col- represented by a n  k matrix. These matrix entries are the
umn exchange method. Procedurally, we exchange q col- discrete units to be exchanged in the LaPSO algorithm.
umns under a minimum-the-best objective function, where Users enter the standard inputs including exchange rates
q ¼ qLB when exchanged with LB SSD and q ¼ qGB when qLB , qGB and maximum number of iterations t in the ini-
exchanged with GB SSD. The deletion step recursively tialization steps.
deletes columns in the candidate SSD that the reduced SSD A major difference between the LaPSO algorithm and
has the smallest Eðs2 Þ values. The addition step adds col- the SIB algorithm is in its updating operation. The MOVE
umns from either LB SSD or GB SSD to the reduced SSD function of LaPSO is actually the MIX operation in the SIB
that the resulting SSD has the smallest Eðs2 Þ values. The algorithm. In specific, assume that a candidate matrix is
MIX operation results in two intermediate SSDs called being exchanged with its personal best. According to Chen
mixwLB and mixwGB. et al. (2013), the standard procedure of LaPSO includes:
The MOVE operation in the SIBSSD algorithm decides (1) randomly choose a position in a column of the candi-
the update direction by comparing the candidate SSD with date LHD; (2) find out the target value in the corresponding
mixwLB and mixwGB. If the Eðs2 Þ of mixwGB is smaller position of the personal best; (3) search the column of the
than the other two SSDs, the SSD is updated as mixwGB, candidate LHD and locate the position possessing the tar-
and so as mixwLB. If the candidate SSD is better than the get value; and (d) swap the values in the two positions.

123
 F. K. H. Phoa

From the language of the SIB algorithm, it is indeed the search Eðs2 Þ-optimal supersaturated designs and /P -opti-
MIX operation that exchanges some entries between the mal Latin hypercube designs.
candidate LHD and the LB or GB LHD. The application of the SIB algorithm can be extended to
Another major difference between two algorithms is that the search of many other classes of matrices and designs.
the LaPSO algorithm does not have a decision-making step For example, one may consider searching for D-optimal
like the MOVE operation. They simply perform the update factorial designs with a defined dimension, or an orthogo-
no matter if the objective function gets better or not. nal array with a defined dimension and a given strength. In
Therefore, it is no guarantee that the update will lead to an fact, the application of the SIB algorithm should not be
equivalent or better LHD in every iteration. The LaPSO limited only to the problems in experimental designs, but
algorithm consists of a RandSwap function that is analog to any problems with discrete domains. One may consider
the MIX operation between the candidate LHD and some using the SIB algorithm to find the best subset of signifi-
random entry choices. Instead of applying it when the LHD cant variables among hundred of variables in a screening
falls into a local attractor/optimum, the LaPSO triggers this experiment for example. Another example is to find the
restart function via a small probability. To the extreme, it optimal locations of change points that leads to the best fit
may restart a candidate LHD that is only one-step close to in a truncated regression model.
the global optimum. There exist some adaptations of PSO for discrete opti-
Although the LaPSO algorithm has several fundamental mization like shufed frog-leaping algorithm Eusuff et al.
difference from the general SIB algorithm, it is still a (2006) and many others. One fundamental difference
successful and fast-converging method for searching /P - between the SIB and PSO frameworks is in the relation-
optimal LHD. We believe that the convergence towards the ships of particles. In PSO or its adaptations, there is a
optimum will arrive even faster with a proper implemen- geometric relationship between particles. For example, if a
tation of the SIB algorithm, and it will be considered as a particle is restricted to be an integer, then it can be a dis-
potential future project. Readers who are interested in crete problem in PSO on a one-dimensional line. This
detailed results of LaPSO can refer to Chen et al. (2013). relationship is required in PSO because of the the updates
of velocity, which is in a certain form of location differ-
ence. In SIB, this geometric relationship is unnecessary
6 Discussion and conclusion because the MIX operation alters the particles by editing its
component units. These units are not limited to numerical
This paper introduces a new natural heuristic method called values but anything that can be evaluated by the objective
the ‘‘Swarm Intelligence Based’’ (SIB) algorithm for the function.
discrete optimization problems. Contrary to the traditional There may be a potential limitation of the SIB method,
PSO algorithm, the SIB algorithm allows users to search the MIX operation in specific, in some discrete problems
for the optimum solutions in a discrete domain. It consists that do not allow a reduction or expansion of the particle
of a MIX operation that combines the candidate particles size. When such situation happens, an alternative operation
with the LB or GB particles, and a MOVE operation that is required and it is still under investigation. Another
selects from three potential choices and makes an update in possible solution is to select a pair of units, which consist
the candidate particle. It also includes a mechanism to of a deleted unit from the current particle and an added unit
avoid the particle being trapped in a local attractor. from the best particle, to swap. Then the particle size is
The SIB algorithm is applied to the search for circulant maintained.
partial Hadamard matrices and the goal is to maximize the On the other hands, SIB can be used to handle opti-
number of columns of the matrices. Due to its circulant mization problem for continuous domain like PSO. Then
property, such matrices are simply represented as generator the particles of SIB being used are all numeric. The pool of
vectors and the algorithm SIBCPHM follows the spirit of available particles in the initialization step and during the
the SIB algorithm and performs entry exchanges on these random jump becomes infinite in size, but the required
generator vectors. As a result, SIBCPHM successfully memory is not affected since SIB never generates or stores
obtains the circulant partial Hadamard matrices with the pool in advance. The MIX and MOVE operations are
maximum number of columns listed in Craigen et al. domain-free and the evaluation of objective function
(2013) when n  52, and even better, the SIBCPHM should be simple. If a density or distribution is known in
algorithm is able to obtain larger matrices with n [ 52. In prior, it can be cooperated in the random selection during
addition to the demonstration of circulant partial Hadamard the initialization and random jump to enhance the search
matrices, the SIB algorithm is available to be applied for speed.

123
A Swarm Intelligence Based (SIB) method for optimization in designs of experiments

When the search domain becomes unexpectedly large, Chakrabarti R, Chattopadhyay PK, Basu M, Panigrahi CK (2006)
the search via the SIB algorithm will undoubtedly require Particle swarm optimization technique for dynamic economic
dispatch. J Inst Eng (India) 87:48–54
more computing effort and parallel computing with graphic Chen RB, Hsieh DN, Hung Y, Wang W (2013) Optimizing Latin
parallel units (GPUs). Specifically, all computations of hypercube designs by particle swarm. Stat Comput 23:663–676
each particle is run in one GPU thread and a synchro- Clerc M (2006) Particle swarm optimization. ISTE, London
nization through all GPU threads at each step is built via a Craigen R, Faucher G, Low R, Wares T (2013) Circulant partial
Hadamard matrices. Linear Algebra Appl 439:3307–3317
separate Compute Unified Device Architecture (CUDA) Dawkins R (1976) The selfish gene. Oxford University Press, London
kernel. Such new technology greatly reduces the compu- Dorigo M (1992) Learning and Natural Algorithms. PhD thesis,
tational times, even though the resulting speedup is still Politecnico di Milano, Italy
case and parameter dependent. The parallel computing Eberhart R, Shi Y (2001) Particle swarm optimization: developments,
application and resources. In: IEEE proceedings of the congress
version of the SIB algorithm is still under development and on evolutionary computation 1:81–85
it will be available upon request from the author in the near Engelbrecht AP (2005) Fundamentals of computational swarm
future. intelligence. Wiley, Chichester
The search results by SIBCPHM and SIBSSD can be Eusuff M, Lansey K, Pasha F (2006) Shuffled frog-leaping algorithm:
a memetic meta-heuristic for discrete optimization. Eng Optim
found in their corresponding download pages in the website 38:129–154
‘‘http://phoa.stat.sinica.edu.tw:10080’’. The search is a Fraser A (1957) Simulation of genetic systems by automatic digital
continuing project and updates are consistently provided computers. I. Introduction. Aust J Biol Sci 10:484–491
when new results are found. Practitioners who need a Geramita A, Seberry J (1979) Quadratic forms, orthogonal designs,
and Hadamard matrices, Lecture notes in pure and applied
design (SSD or CPHM) of their required dimensions can mathematics series 45, Marcel Dekker Inc., New York
enter the corresponding information and a design is sent via Gong T, Tuson AL (2008) Particle swarm optimization for quadratic
email in return. It is a convenient tool for most practitioners assignment problems—a forma analysis approach. Int J Comput
who use the designs as their experimental plans and are Intell Res 4:177–185
Horadam KJ (2007) Hadamard matrices and their applications.
without expert knowledge in the underlying theorems in Priceton University Press, Princeton
design of experiments. Kennedy J, Eberhart R (1995) Particle swarm optimization. In:
Proceedings of IEEE international conferenec on neural net-
Acknowledgments This work is supported by Career Development works 4:1942–1948
Award of Academia Sinica (Taiwan) grant number 103-CDA-M04 Low RM, Stamp M, Craigen R, Faucher G (2005) Unpredictable bi-
and Ministry of Science and Technology (Taiwan) grant numbers nary strings. Congr Numer 177:65–75
102-2628-M-001-002-MY3 and 104-2118-M-001-016-MY2. Morris MD, Mitchell TJ (1995) Exploratory designs for computa-
tional experiments. J Stat Plan Inference 43:381–402
Orouskhano M, Orouskhani Y, Mansouri M, Teshnehlab M (2013) A
novel cat swarm optimization algorithm for unconstrained
References optimization problems. I J Inf Technol Comput Sci 11:32–41
Panduro MA, Brizuela CA, Balderas LI, Acosta DA (2009) A
Bera R, Roy JS (2013) Thinning of elliptical and concentric elliptical comparison of genetic algorithms, particle swarm optimization
antenna arrays using particle swarm optimization. Microw Rev and the differential evolution method for the design of scannable
19:2–7 circular antenna arrays. Prog Electromagn Res B 13:171–186
Box KHV, Cox DR (1962) Some systematic supersaturated designs. Passino KM (2002) Biomimicry of bacterial foraging for distributed
Technometrics 4:489–495 optimization and control. IEEE Control Syst Mag 22:52–67
Boyd S, Vandenverghe L (2004) Convex optimization. Cambridge Phoa FKH, Chen RB, Wang W, Wong WK (2016) Optimizing two-
University Press, Cambridge level supersaturated designs using swarm intelligence tech-
Braendler D, Hendtlass T (2002) The suitability of particle swarm niques. Technometrics 58:43–49
optimization for training neural hardware. In: International Plackett RL, Burman JP (1946) The design of optimum multifactorial
conference on industrial and engineering applications of artificial experiments. Biometrika 33:305–325
intelligence and expert system, 190–199 (2002) Seeley TD (1995) The wisdom of the hive. Harvard University Press,
Brandstatter B, Baumgartner U (2002) Particle swarm optimization— Cambridge
mass-spring system analogon. IEEE Trans Magn 38:997–1000 Shi Y, Eberhart R (1998) A modified particle swarm optimizer. In:
Bulutoglu DA (2007) Cyclicly constructed Eðs2 Þoptimal supersatu- Proceedings of IEEE international conference on evolutionary
rated designs. J Stat Plan Inference 137:2413–2428 computation, 69–73
Bulutoglu DA, Cheng CS (2004) Construction of Eðs2 Þ-optimal Xu H, Phoa FKH, Wong WK (2009) Recent developments in
supersaturated designs. Annal Stat 32:1662–1678 nonregular fractional factorial designs. Stat Surv 3:18–46

123