Overview

WinBUGS: a tutorial
Anastasia Lykou1 and Ioannis Ntzoufras2,

The reinvention of Markov chain Monte Carlo (MCMC) methods and their
implementation within the Bayesian framework in the early 1990s has established
the Bayesian approach as one of the standard methods within the applied
quantitative sciences. Their extensive use in complex real life problems has lead
to the increased demand for a friendly and easily accessible software, which
implements Bayesian models by exploiting the possibilities provided by MCMC
algorithms. WinBUGS is the software that covers this increased need. It is the
Windows version of BUGS (Bayesian inference using Gibbs sampling) package
appeared in the mid-1990s. It is a free and a relatively easy tool that estimates the
posterior distribution of any parameter of interest in complicated Bayesian models.
In this article, we present an overview of the basic features of WinBUGS, including
information for the model and prior specification, the code and its compilation, and
the analysis and the interpretation of the MCMC output. Some simple examples
and the Bayesian implementation of the Lasso are illustrated in detail. 2011 John
Wiley & Sons, Inc. WIREs Comp Stat 2011 3 385396 DOI: 10.1002/wics.176

HISTORICAL BACKGROUND of the parameters of interest in a variety of models

using MCMC. It only requires to specify the model
T he BUGS (Bayesian inference using Gibbs
sampling) project was introduced in 1989 by the
research group of David Spiegelhalter in the MRC
code in which the model likelihood and the prior
distribution are defined. The programming syntax
and requirements are similar to the ones in R
Biostatistics Unit as a software that uses Markov chain
and Splus. For those who are not familiar with
Monte Carlo (MCMC) methods in complex Bayesian
programming, there is also DOODLE, a graphical
methods. BUGS was initially a software for DOS
interface in which we can define our model via
and Linux operating systems limited in specialized
drawing the corresponding directed acyclic graph
algorithms. The project started expanding after it
(DAG). Moreover, WinBUGS is freely available
moved to the Imperial College in 1996 headed by
and this is one of the reasons that contributed
Nicky Best, and the first version of WinBUGS for
in its growing popularity. A detailed manual,2 as
Windows became available in 1997. In the following
well as three volumes of examples35 that provide
years, WinBUGS was improved and extended by
important help and insight to the WinBUGS user,
considering more complicated model structures. Later
is available in the BUGS project website (www.
when Jon Wakefield and Dave Lunn joined the group,
mrc-bsu.cam.ac.uk). A comprehensive introduction
WinBUGS capabilities improved impressively. The
latest version of WinBUGS (1.4.3) was developed in Bayesian modeling using WinBUGS is also offered
jointly by Imperial College and the School of Medicine by Ntzoufras,6 in which emphasis is given on model
at St Marys, London. More details can be found in building, implementation using WinBUGS, and the
Ref 1 and in the web page of OpenBUGS (http://www. interpretation and analysis of the posterior results.
openbugs.info). The purpose of this article is to provide a com-
WinBUGS has become widely popular over the prehensive short tutorial by summarizing the most
last years as it can estimate the posterior distributions important features of WinBUGS. It provides informa-
tion on how to code and compile a model through two
Correspondence to: ntzoufra@aueb.gr simple examples and a more detailed one that illus-
1
Department of Mathematics and Statistics, Lancaster University, trates the implementation of Bayesian Lasso method in
Lancaster, UK WinBUGS. The remainder of this article is organized
2 Department of Statistics, Athens University of Economics and as follows. Section Getting Started with Winbugs
Business, Athens, Greece contains a brief description of the menu bar and
DOI: 10.1002/wics.176 detailed illustration of the model specification, the

Vo lu me 3, September/Octo ber 2011 2011 Jo h n Wiley & So n s, In c. 385

 Overview wires.wiley.com/compstats

data and the initial values definition using two simple random observations from the posterior distribution
examples. Section Running an MCMC in Winbugs and the Monitor Met tool checks the acceptance
and Obtaining Posterior Summaries provides a con- rate of the Metropolis-Hasting algorithm (in cases
densed list of the actions needed to run the MCMC it is used). The last set of generated values can be
algorithm for a model and how to obtain and analyze saved using the Save State tool. This set of values
the corresponding output. An example of implement- can be used as initial values in cases that we want
ing the Bayesian Lasso in WinBUGS can be found in to rerun the MCMC algorithm from the point that a
the Section An Illustrating ExampleImplementing previous MCMC run stopped. The Inference menu
Bayesian Lasso in Winbugs. The article closes with provides information about the posterior distribution
a short discussion concerning the future potentials of of the model parameters. This information is based
WinBUGS followed by a short conclusion. on the analysis of the MCMC output. Under this
menu, the most frequently used tool is the Samples
tool, which provides basic descriptive summaries and
GETTING STARTED WITH WINBUGS graphical representation for specific attributes of the
The Menu Bar MCMC output and the corresponding estimated pos-
terior distribution.
The latest version of WinBUGS (1.4.3) as well
The Info and Options menus are offering
as installation instructions and the free key for
some auxiliary tools for MCMC analysis. From the
unrestricted use can be found on the soft-
former menu, we can open or clear a log window
wares website: www.mrc-bsu.cam.ac.uk/bugs/win
(where WinBUGS results and figures are printed) or
bugs/contents.shtml. Once the installation process
we can extract the current parameter values using
has been completed, WinBUGS can be assessed by
the Node info tool. From the latter menu, we can
double-clicking its shortcut. All main operations are
change some options concerning the output, the block-
available in a menu bar, which is similar to the ones
ing and the generation algorithm used to update each
found at any windows-based program.
parameter.
The basic operations in the menu bar are
Finally, the Map and the Doodle menus are
the File, Window, and Help menus. The File
offering more specialized tools for the user. The Map
menu handles the usual file actions that are
menu corresponds to the GeoBUGS add-in module
available in any windows-based software (e.g., Open,
and it can be used for spatial modeling and mapping.
Close, Save, Print), the Window manages
The Doodle menu is used to construct the DAG
active windows in WinBUGS and the Help provides
that describes the conditional dependencies of the
access to the detailed manual and examples of
Bayesian model we wish to fit. This tool is essential
WinBUGS which are very useful for the user.
for those who are not familiar with programming as
The Tools, Edit, Attributes, and Text
the model code can be generated from this graphical
menus refer to editing facilities of the documents in
representation of the model. Nevertheless, it assumes
WinBUGS. To be more specific, the Edit menu allows
very good knowledge and understanding of Bayesian
the user to do the usual editing actions of any word-
models and how they can be represented in terms of
processing software (e.g., Copy, Cut, Paste),
conditional distributions and hierarchies.
while the Tools menu manages more specialized
actions available for WinBUGS compound doc-
uments (insert dates, encode, and decode How to Code a Model
algorithms). The WinBUGS user can change the WinBUGS uses its own type of input and output files
color, the font type, and size of the text in a com- that are called compound documents and are saved
pound document using the Attributes menu, while with the odc suffix. The first step in WinBUGS is
he/she can find or replace a text, insert a ruler, a para- to specify the model, which includes the likelihood
graph, or blank spaces (amongst other operations) in function for the observed sample and the prior
the Text menu. information for the parameters. The model code is
The most substantial WinBUGS operations are written in a compound file within the syntax
included in the Model and Inference menus. model { ...... } .
They include all the commands and actions related There are three categories for the model
to running the MCMC algorithm and analyzing its parameters (or nodes): the constant, the stochastic,
output in order to obtain posterior results. or random and the logical. The constant nodes are
The Specification tool, under the Model fixed values while the stochastic are random variables,
menu, is used to compile the model and initialize such as the data and the model parameters. The
the MCMC algorithm; the Update tool generates stochastic nodes follow a distribution, which can be

386 2011 Jo h n Wiley & So n s, In c. Vo lu me 3, September/Octo ber 2011

 WIREs Computational Statistics WinBUGS

specified by using commands similar with the ones A[i,j,k] correspond to the Aijk element. Arrays
used in R and Splus packages. The logical nodes of lower dimensions or parts of the initial array can
are mathematical expressions of other (constant or be derived using syntax similar to the one used for the
stochastic) components. matrices; further details can be found in Section 3.3.2
of Ref 6.
Example 1 Calculations among vectors, matrices, and
Lets assume that a part of a Bayesian model includes arrays cannot be performed directly in WinBUGS.
a normally distributed variable X Normal(, 2 = The following calculation xT y between the vectors
1/ ), with known mean = 0 and unknown x and y can be performed using the function
precision . If the uncertainty about the precision inprod( x[], y[] ) and the multiplication
is expressed by considering the Gamma prior between two matrices A and B of dimension n k
distribution Gamma(0.01, 0.01), this can be and k m can be performed using the inprod for all
expressed in WinBUGS as follows: the rows and columns. Thus, the use of the function
for is required to define a loop among the rows and
model{ the columns, and the multiplication is performed as
mu <- 0 follows:
a<-0.01
b<-0.01 for (i in 1:n){
X ~ dnorm( mu, tau ) for (j in 1:m){
tau ~ dgamma( a, b ) C[i,j] <- inprod
sigma2 <- 1/tau # sigma2: (A[i,],B[,j])
variance of the normal }
distribution }
}
A set of built-in functions are available within
In the above syntax, the sign # is used to add WinBUGS including arithmetic functions such as the
comments, ~ is used to specify that a random absolute value (abs), the exponential (exp), the
node follows a distribution and <- is used to natural logarithm (log), the square root (sqrt), and
define assignments for constant and logical nodes. the statistical functions such as the sum (sum) the
Nodes mu, a, b are constants here, X and tau standard deviation (sd) and the rank (rank). A list
are stochastic components and sigma2 is a logical with the functions that can be used in WinBUGS can
component. The commands dnorm( mu, tau) and be found on the softwares manual.2
dgamma(a,b) are used to specify the normal (with
mean mu and variance 1/tau) and the gamma (with Model Specification
mean a/b) distributions respectively. A list with all Suppose that n realizations are available for the
the distributions available in WinBUGS can be found response variable y, which follows a distribution with
at the softwares manual2 and in Ref 6, p. 9091. parameter vector . Assume that the parameter is
Each component/node must be uniquely defined in the related with the explanatory variables X1 , X2 , . . . , Xp
WinBUGS syntax. through a link function h and a parameter vector .
Vectors, matrices, and arrays can be represented The prior information for is expressed by a known
in WinBUGS in a similar way as in R/Splus pack- distribution. This model is specified in WinBUGS with
ages. In the example above, the random variable X is the following syntax.
a scalar. If X is a n-dimensional random vector this is
denoted in WinBUGS by X[]. Syntax X[i] refers to # Likelihood
the i-th element of vector X for i = 1, . . . , n, whereas, for (i in 1:n){
X[i:j] extracts a vector with components the i-th y[i] ~ distribution.name(theta)
up to the j-th element of X. If M is a matrix this is }
specified in WinBUGS by M[]. If M is a n p matrix # Link function
then M[i,j] corresponds the mij element of M for any theta <- [function of beta
i {1, . . . , n} and j {1, . . . , p}. The entire i-th row can and Xs]
be extracted by typing M[i,] while the j-th column by # prior distribution
M[,j]. The sub-matrix that contains all elements mij beta ~ distribution.name( ... )
with i1 i i2 and j1 j j2 can be extracted using
the syntax M[i1:i2,j1:j2]. A three-dimensional If the parameters , have dimension higher than one,
array A is denoted by A[ , , ] and the item a for loop is needed to specify them.

Vo lu me 3, September/Octo ber 2011 2011 Jo h n Wiley & So n s, In c. 387

 Overview wires.wiley.com/compstats

Example 2 for (i in 1:n){

Consider a normal linear regression model with y[i] ~ dnorm( mu[i], tau )
the response vector y = (y1 , . . . , yn )T following the mu[i] <- inprod(X[i,]) ,
normal distribution and xi1 , xi2 the realizations beta[] )
of 2 explanatory variables X1 , X2 for i = 1, . . . , n }
individuals. The Bayesian model is formulated as # prior distributions
for (j in 1:p+1){
Yi N(i , 2 ) for i = 1, . . . , p beta[j] ~ dnorm( 0.0 ,
1.0E-4 )
and the parameter i is a linear combination of the }
explanatory variables, tau ~ dgamma( 0.01, 0.01 )
# other parameters of interest
i = 0 + 1 x1i + 2 x2i . s2 <- 1/tau
s <- sqrt(s2)
The Normal distribution is chosen to express the prior }
information for and the Gamma distribution for the
precision = 2 . Hence,
Data and Initial Value Specification
i Normal(0, 104 ) for i = 0, 1, 2,
To complete the model specification in WinBUGS, we
Gamma(0.01, 0.01). need to insert the data and provide some initial values
for the MCMC algorithm.
The syntax of the above regression model The initial values are used as a starting point
follows. in the iterative procedure of the MCMC algorithm.
Initial values are needed for all the model parameters,
model{
i.e., stochastic components of the model that are not
# definition of the likelihood
data and for which prior distributions are imposed.
function
The option of random generation of a part or all
for (i in 1:n){
the initial values is available in WinBUGS but it is
y[i] ~ dnorm( mu[i], tau )
advised to be avoided as it may lead to bad starting
mu[i] <- beta0 + beta1*
points entailing slow mixing of the algorithm or even
x1[i] + beta2 * x2[i]
numerical errors such as overflows.
}
The syntax used to initialize values for the model
# prior distributions
in Example 2 is
beta0 ~ dnorm( 0.0 , 1.0E-4 )
beta1 ~ dnorm( 0.0 , 1.0E-4 ) # INITS
beta2 ~ dnorm( 0.0 , 1.0E-4 ) list( beta=0, beta1=0, beta2=0,
tau ~ dgamma( 0.01, 0.01 ) tau=1 )
# other parameters of interest
s2 <- 1/tau The data can be written using the list syntax
s <- sqrt(s2) which is similar to the corresponding one used in
} R/Splus. For example, the data of the Example 2
with 2 explanatory variables of n = 5 observations
This syntax can be written in a more general way can be specified as follows:
in order to cover the case with p explanatory variables.
This can be achieved by introducing the parameter # DATA
vector = (0 , 1 , . . . , p )T and the n (p + 1) data list(n=5,
matrix X = (1n , X 1 , . . . , X p ), where 1n is a vector x1=c(1,2,3,4,5),
of length n with all elements equal to one and X j x2=c(23,57,59,14,36)
(j = 1, . . . , p) is the vector of length n, with elements )
the observed values of covariate Xj . The syntax can Alternatively, the data can be specified in matrix
be now written as follows: form as follows:
model{ # DATA
# definition of the likelihood list(n=5, p=2,
function X = structure(.Data=c(1,23,2,

388 2011 Jo h n Wiley & So n s, In c. Vo lu me 3, September/Octo ber 2011

 WIREs Computational Statistics WinBUGS

57,3,59,4,14,5,36) 3. If steps 1 and 2 are successful, then the model is

.Dim=c(5,2)) compiled by clicking the corresponding button.
Both codes above define the same data matrix but with The message model compiled should appear.
different types of nodes. The second code is preferable If an error message is produced, then the model
when the number of variables involved is large. It code must be corrected and the procedure must
is also advised to use R and the command dput to be started from step 1 again.
directly import the data in WinBUGS with this format; 4. Once the model is compiled successfully, the
see Section 3.4.6.2 in Ref 6 for more details. load inits button becomes active. High-
Alternatively, the data can be represented in a light the word list that contains the ini-
simpler, rectangular form. The names of the vector tial values of the stochastic components and
nodes (followed by []) must be stated in the first row click the load inits button. The button
followed by the variable values for each observation in gen inits can alternatively be used to gen-
each line. The data must conclude with the command erate random initial values for the parame-
END followed by a blank line. All variablesvector ters. If the message this chain contains
nodes must have the same length. uninitialized variables appears, then
DATA some stochastic nodes do not have initial val-
y[] x1[] x2[] x3[] ues. In such cases gen inits can be also
20 23 1 14 used. If the message model is initialized
19 6 0 23 appears in the left bottom bar of WinBUGS then
32 18 1 12 the MCMC is ready to start generating random
22 20 1 12 values.
... ... ... ... 5. Open the Update tool under the Model menu
END and specify the number of iteration in the
(Blank line) burn-in period in the box entitled updates.
The number in the box refresh shows how
often the results are refreshed, thin defines the
RUNNING AN MCMC IN WINBUGS lag between stored iterations and iteration
AND OBTAINING POSTERIOR shows the current number of iterations of the
SUMMARIES MCMC algorithm. Insert the desired numbers
and click update. The iteration counter will
Once the model, the data, and the initial values have
been specified we can compile and run the MCMC start changing until the required number of
algorithm. The procedure to be followed is described iterations is reached.
below. 6. Open the Sample Monitor tool under the
Inference menu and select the parameters for
1. Open the Specification tool under the which we wish to infer about. Write the name of
Model menu and highlight the word model. the parameters in the node box and press set.
WinBUGS checks the syntax of the model by This should be repeated for all the parameters
clicking the check model button. A message that we wish to monitor. Return to the update
appears in the bottom left-hand corner of tool and select the number of iterations that we
the window indicating whether the model is wish to generate and then press the updates
syntactically correct or not. If an error message button. When the counter reaches the required
appears, the model code must be corrected and number of iterations then an MCMC sample
revised and then the model syntax must be from the posterior distribution is available for
checked again (note that the cursor indicates the monitored variables.
the location of the error in the code).
2. If the model is syntactically correct, the load Analysis of the MCMC output and inference
data box becomes accessible. Highlight the concerning the posterior distribution can be obtained
word list, if the data are presented in the by the Sample Monitor tool. Summaries of the
list form, or the first line of the rectangular posterior distributions can be derived by typing the
form and press the load data button. The name of the parameter of interest in the node box (or
message data loaded should appear. If more using the pull down menu) and pressing the stats
than one dataset is available we repeat the above button. The posterior summaries of all monitored
procedure until all the data are loaded. parameters can be extracted if we use the asterisk

Vo lu me 3, September/Octo ber 2011 2011 Jo h n Wiley & So n s, In c. 389

 Overview wires.wiley.com/compstats

( ) in the node box. A density plot of the marginal Example

posterior distribution of each node can be obtained We illustrate the Bayesian Lasso regression using a
using the density button. simulated data set that consists of n = 50 observations
The remaining buttons can be used to obtain and p = 5 covariates generated from independent
a basic analysis of the MCMC output by checking standardized normal distributions and the response
the convergence of the chain (history), the Monte from
Carlo (MC) error (stats) and the autocorrelation
(auto cor). The thinning interval of observations Yi Normal(Xi3 + Xi4 , 2.52 ), for i = 1, . . . , 50.
used to derive the summaries can be controlled by the
corresponding box. A detailed diagnostic analysis of The Bayesian Lasso is performed on this dataset for
the MCMC output can be conducted in R, using the different values of by considering 2, 000 updates
CODA7 and BOA8 packages. The observations of the after discarding additional 1,000 iterations as burn-in
posterior distributions of the WinBUGS output can period.
be obtained in a compatible format using the option In the Bayesian implementation, we standardize
coda. both the response and the covariates. For this reason,
the constant parameter is not included in the model
for the standardized variables as it is natural to set
AN ILLUSTRATING EXAMPLE
it equal to zero. All the variables are standardized
IMPLEMENTING BAYESIAN LASSO within the WinBUGS code (see lines 214 in the
IN WINBUGS code of Algorithm 1). Note that WinBUGS allows
The Lasso9 is a shrinkage and variable selection to define stochastic nodes twice only in the case of
method for normal linear regression models. Its transforming response variables as in this example.
estimates are obtained by imposing the L1 norm The original regression parameters are calculated in
penalty on the linear regression coefficients. Owing to WinBUGS using simple logical nodes as
the shape of the L1 norm, the regression coefficients
are shrunk toward zero and some of them are set 
p
0 = y Sy bj xj
exactly equal to zero.
j=1
The Lasso estimates have an apparent Bayesian
perspective as they correspond to the posterior Sy
j = bj
mode of a Bayesian normal linear regression model Sxj
using a double-exponential prior distribution for the
regression coefficients. The corresponding model is 2 = S2y z2
described by the following formulation:
where z2 is the error variance, bj the coefficients of
1
Yi |, Normal(i , ), for i = 1, . . . , n the regression model using the standardized variables,
y and xj the sample means, and S2y and S2xj the sample
= X with = (1 , 2 , . . . , n )T

 variances for Y and Xj respectively.
1 The code for the above-defined Bayesian Lasso
j DE 0, , for j = 1, . . . , p,
model is given in Algorithm 1. The model was
Gamma(a, d), initialized by setting all standardized coefficients equal
to zero and the corresponding precision equal to one.
where = 1/ 2 is the precision of the Normal Following the arguments in Ref 10, we choose the
regression model and is the shrinkage parameter value of = 0.067.
which controls the prior variance given by 2/( )2 . We follow the procedure described in
Different values of correspond to different levels Section Running an MCMC in WinBUGS and
of shrinkage. Small values of correspond to large Obtaining Posterior Summaries to obtain summaries
variance introducing low information in the Bayesian for the posterior distribution. The posterior sum-
model and essentially not imposing any shrinkage maries of the parameters of the Lasso regression
on the model parameters. On the other hand, large coefficients and the variance of the regression model
values of correspond to strong prior that is close are displayed in Figure 1 for the standardized data and
to zero resulting to high levels of shrinkage. To ensure in Figure 2 for the original data. The posterior means
that has the same meaning for all covariate effects, and medians of the coefficients of X1 and X2 are
we assume, without loss of generality, that all the very low indicating that these are the least important
variables have mean zero and variance equal to one. variables. As expected, the variables used to generate

390 2011 Jo h n Wiley & So n s, In c. Vo lu me 3, September/Octo ber 2011

 WIREs Computational Statistics WinBUGS

ALGORITHM 1 | WinBUGS Code for the Bayesian Lasso Model.

the response (X3 , X4 ) have the highest posterior mea- of the coefficients obtained using the two values of
sures, whereas the last covariate (X5 ) has moderate are depicted in the box plots of Figure 5. The most
measures. Similar conclusions are drawn from the noteworthy change is observed for 5 , for which zero
Figure 3, which gives the densities of the posterior lies outside the 95% posterior credible interval in the
summaries of the Lasso coefficients. The results here first run and inside this interval in the second run.
provide some evidence about the important variables, We can extend the aforementioned model in
although the variable selection problem is not directly order to incorporate the variable selection procedure
addressed. Moreover, we observe that the posterior in our formulation. This can be achieved by
means of
 are similar to the ordinary least square esti-
 introducing a vector of binary indicators that
mates  = (0.16, 0.19, 0.067, 1.19, 1.67, 1.11)T highlights which variables are included in the model
concluding that our prior was essentially noninforma- (with j = 1) or not (with j = 0) as in Refs 11 and 12.
tive implementing minor (or no) shrinkage on the This formulation was proposed by Lykou and
model parameters. For illustration, we also rerun Ntzoufras10 and is described below.
the model with = 2 resulting to a posterior with
summaries given in Figure 4. For this value of , the
Yi |, , Normal(i , 1 ), for i = 1, 2, . . . , n
posterior means are shrunk by 42% for 1 and from
( ) ( ) ( )
5 to 20% for the rest of the coefficients. Differences = X ( ) with ( ) = ( , , . . . , )T
1 2 p

Vo lu me 3, September/Octo ber 2011 2011 Jo h n Wiley & So n s, In c. 391

 Overview wires.wiley.com/compstats

1_node_stats_standardized

node mean sd MC error 2.5% median 97.5% start sample

b[1] 0.0505 0.1021 0.002783 0.2448 0.05336 0.152 1001 2000
b[2] 0.02172 0.09605 0.00228 0.1711 0.02279 0.2092 1001 2000
b[3] 0.3659 0.09715 0.002324 0.1757 0.3666 0.5611 1001 2000
b[4] 0.5782 0.09821 0.002123 0.3859 0.5757 0.7774 1001 2000
b[5] 0.2293 0.1006 0.002069 0.4231 0.2312 0.03014 1001 2000
sz 0.6609 0.06534 0.001564 0.5464 0.6557 0.8078 1001 2000

FIGURE 1 | Posterior summaries of the Lasso regression parameters using standardized data ( = 0.067).

2_node_stats_full

node mean sd MC error 2.5% median 97.5% start sample

beta0 0.1655 0.1622 0.0032 0.1563 0.1688 0.4795 1001 2000
beta[1] 0.1889 0.382 0.01041 0.9156 0.1996 0.5686 1001 2000
beta[2] 0.07233 0.3199 0.007593 0.5699 0.07589 0.6966 1001 2000
beta[3] 1.19 0.3158 0.007557 0.5714 1.192 1.824 1001 2000
beta[4] 1.668 0.2834 0.006126 1.113 1.661 2.243 1001 2000
beta[5] 1.103 0.4837 0.009954 2.035 1.112 0.145 1001 2000
sigma 2.197 0.2173 0.0052 1.817 2.18 2.686 1001 2000

FIGURE 2 | Posterior summaries of the Lasso regression parameters for the unstandardized data ( = 0.067).

3_densities

beta0 sample: 2000 beta[1] sample: 2000

3.0 1.5
2.0 1.0
1.0 0.5
0.0 0.0
0.5 0.0 0.5 2.0 1.0 0.0 1.0

beta[2] sample: 2000 beta[3] sample: 2000

1.5 1.5
1.0 1.0
0.5 0.5
0.0 0.0
0.2 1.0 0.0 1.0 0.0 1.0 2.0

beta[4] sample: 2000 beta[5] sample: 2000

1.5 1.0
1.0 0.75
0.5
0.5 0.25
0.0 0.0
0.0 1.0 2.0 3.0 2.0 1.0 0.0

FIGURE 3 | Posterior densities of the Lasso regression coefficients for the unstandardized data ( = 0.067).

392 2011 Jo h n Wiley & So n s, In c. Vo lu me 3, September/Octo ber 2011

 WIREs Computational Statistics WinBUGS

( ) constrained to zero if Xj is not included in the model

j = j j
formulation.
 1
j | DE 0, , for j = 1, . . . , p, In order to extend the model code of Algorithm 1
according to the above-mentioned formulation we
Gamma(a, d), need to
j Bernoulli(j ) 1. substitute line 18 with the following syntax
where Bernoulli( ) is the Bernoulli distribution with mu[i] <- inprod(Z.X[i,],
success probability . The actual model parameters bgamma[])
( ) are denoted with a vector of length p with
( )
elements j = j j (for j = 1, . . . , p) which are in order to substitute j by j bj ,

4_stats_lambda2

node mean sd MC error 2.5% median 97.5% start sample

b[1] 0.0291 0.09658 0.002671 0.2226 0.0229 0.1582 1001 2000
b[2] 0.01712 0.09101 0.002357 0.1641 0.01255 0.2111 1001 2000
b[3] 0.3292 0.1033 0.002385 0.1267 0.3278 0.5392 1001 2000
b[4] 0.5424 0.1067 0.002194 0.3387 0.541 0.7633 1001 2000
b[5] 0.1876 0.1063 0.002806 0.4028 0.1829 0.006072 1001 2000
beta[1] 0.1089 0.3613 0.009992 0.8328 0.08568 0.5918 1001 2000
beta[2] 0.05702 0.3031 0.007851 0.5466 0.04179 0.703 1001 2000
beta[3] 1.07 0.3359 0.007752 0.4119 1.066 1.753 1001 2000
beta[4] 1.565 0.3078 0.006331 0.9774 1.561 2.202 1001 2000
beta[5] 0.9025 0.5115 0.0135 1.938 0.88 0.02921 1001 2000
beta0 0.2077 0.167 0.003606 0.131 0.2079 0.5444 1001 2000
sigma 2.408 0.2394 0.006507 1.999 2.385 2.93 1001 2000
sz 0.7242 0.07201 0.001957 0.6013 0.7174 0.8811 1001 2000

Standardized data: b [ j ] = bj ; sz = sz
Unstandardized data: beta = b0 (constant term); beta [ j ] = bj ; sigma = s

FIGURE 4 | Posterior summaries of the Lasso regression parameters for standardized and unstandardized data ( = 2).

(a) (b)

5b_boxplot_lambda067 5_boxplot_lambda2

box plot: b box plot: b

1.0 1.0
[4] [4]

[3] [3]

0.5 0.5

[2]
[2] [1]
[1]
[5]

0.0
[5] 0.0

0.5
0.5

FIGURE 5 | Posterior box plots describing the 95% credible intervals of the regression coefficients using the standardized data (obtained by
inserting node b in the node box of Compare tool inside the Inference menu.). (a) = 0.067; (b) = 2.

Vo lu me 3, September/Octo ber 2011 2011 Jo h n Wiley & So n s, In c. 393

 Overview wires.wiley.com/compstats

2. substitute line 33 by X3 and X4 remained almost unchanged (shrunk by

8.6 and 0.06% respectively). The posterior inclusion
beta0<- meany - sdy * probabilities for each covariate can be estimated from
inprod( bgamma[], meansd[] ) the posterior means of each j as given in Figure 7.
From the results, we clearly observe that covariates X3
and X4 must be included in the model having posterior
in order to calculate correctly the constant probabilities 0.9 and 1.0, respectively. Covariates
parameter for any given model, X1 and X2 have very low posterior probabilities
3. add the following syntax ( 1.8%), while covariate X5 is included in the model
with posterior probability of 17%.
for (j in 1:p){ Figure 8 displays the densities of the posterior
# b x gamma for the distributions of ( ) . The posterior distributions of
standardized data the coefficients for the non-important variables have
bgamma[j] <- b[j] * high spikes at zero except for covariate X4 , which
gamma[j] is the most important one in this illustration and
# prior for gamma its posterior distribution is well placed away from
gamma[j]~dbern(0.5) zero. The posterior distribution of 3 is bimodal,
# beta*gamma for the with one mode at zero and one at a positive value.
unstandardized data However, the posterior median of the distribution
betagamma[j] <- beta[j] * and the posterior mean of the corresponding inclusion
gamma[j] probability (given by the mean of 3 and it is equal
} to 0.90) indicate that this variable should also be
included in the model.

in order to define bj = j bj , j = j j and the
prior for each j . DISCUSSION: A FUTURE FULL
OF PROMISES
Posterior summaries for ( ) are given in Over the last years WinBUGS gained a tremendous
Figure 6 for = 0.067. We clearly observe that popularity within the scientific community. WinBUGS
the first two coefficients are shrunk toward zero has been used in a wide range of practical problems
(the posterior means shrunk by 99.6 and 96% having different scientific backgrounds. It has been
respectively). Also the posterior distribution of 5 the object of interest in various short courses and
was dramatically pushed toward zero with the workshops. More and more universities include
posterior mean shrunk by 83.7%. On the other Bayesian data analysis courses using WinBUGS in
hand, the posterior means for the coefficients of their syllabus; see BUGS resources online link in the

6_stats_gamma

node mean sd MC error 2.5% median 97.5% start sample

beta0 0.2324 0.1997 0.009046 0.08211 0.2041 0.7027 1001 10000
betagamma[1] 7.3E-4 0.05531 5.839E-4 0.0 0.0 0.0 1001 10000
betagamma[2] 0.002618 0.04639 5.232E-4 0.0 0.0 0.0 1001 10000
betagamma[3] 1.087 0.4784 0.02172 0.0 1.155 1.844 1001 10000
betagamma[4] 1.669 0.2978 0.003231 1.076 1.669 2.251 1001 10000
betagamma[5] 0.1804 0.4392 0.01275 1.541 0.0 0.0 1001 10000
sigma 2.39 0.264 0.00703 1.941 2.367 2.98 1001 10000
bgamma[1] 1.951E-4 0.01479 1.561E-4 0.0 0.0 0.0 1001 10000
bgamma[2] 7.862E-4 0.01393 1.571E-4 0.0 0.0 0.0 1001 10000
bgamma[3] 0.3343 0.1472 0.006681 0.0 0.3552 0.5672 1001 10000
bgamma[4] 0.5783 0.1032 0.00112 0.373 0.5783 0.78 1001 10000
bgamma[5] 0.0375 0.09131 0.002651 0.3204 0.0 0.0 1001 10000
sz 0.7188 0.07941 0.002114 0.5837 0.7118 0.8963 1001 10000

Standardized data: bgamma [j] = gj bj ; sz = sz

Unstandardized data: beta = b0 (constant term); betagamma [j] = gj bj ; sigma = s

FIGURE 6 | Posterior summaries of the Lasso regression coefficients with variable selection.

394 2011 Jo h n Wiley & So n s, In c. Vo lu me 3, September/Octo ber 2011

 WIREs Computational Statistics WinBUGS

6_stats_gamma2

node mean sd MC error 2.5% median 97.5% start sample

gamma[1] 0.0181 0.1333 0.001983 0.0 0.0 0.0 1001 10000
gamma[2] 0.0178 0.1322 0.001988 0.0 0.0 0.0 1001 10000
gamma[3] 0.9016 0.2979 0.01782 0.0 1.0 1.0 1001 10000
gamma[4] 1.0 0.0 1.0E-12 1.0 1.0 1.0 1001 10000
gamma[5] 0.173 0.3782 0.01213 0.0 0.0 1.0 1001 10000

FIGURE 7 | Posterior summaries the indicator parameters included in the Bayesian Lasso model.

7_densities_gamma

beta0 sample: 10000 betagamma[1] sample: 10000

3.0 150.0
2.0 100.0
1.0 50.0
0.0 0.0
0.5 0.0 0.5 1.0 0.0 1.0

betagamma[2] sampe: 10000 betagamma[3] sample: 10000

150.0 1.5
100.0 1.0
50.0 0.5
0.0 0.0
1.0 0.0 1.0 1.0 0.0 1.0 2.0

betagamma[4] sampe: 10000 betagamma[5] sample: 10000

1.5 15.0
1.0 10.0
0.5 5.0
0.0 0.0
0.0 1.0 2.0 3.0 2.0 1.0 0.0

FIGURE 8 | Posterior densities for model parameters ( ) .

WinBUGS website for a coherent list of such activities systems of ordinary differential equations and the
and courses. WinBUGS jump interface to perform variable selection
The popularity of WinBUGS has motivated through the Reversible Jump MCMC.14,15
various expansions and made WinBUGS applicable in WinBUGSs popularity has also been extended
a wider range of disciplines, such as social, actuarial
due to its compatibility with other statistical packages
science, population genetics, and archaeology.
and software. WinBUGS runs from GenStat, which is
GeoBUGS developed by the team in the Imperial
a software for bioscience. It can also be called via R
College at St Marys Hospital can be used to fit
spatial models and produce a range of maps as output. using the R2WinBUGS R-library on Comprehensive
PKBUGS has been developed by Dave Lunn to fit R Archive Network (CRAN) site. There are also
pharmacokinetic models. The WinBUGS development codes that can be used to call WinBUGS through
interface (WBDev)13 allows the users to define their Stata,16 SAS and a code for Excel that does not
own distributions and functions. This was originally require any knowledge of WinBUGS. MATBUGS is
designed for social scientists but it has been used by a Matlab interface for WinBUGS, which has been
other researchers as well. Lunn has also developed extended to run in Linux systems. Details about it can
the WinBUGS Differential Interface (WBDiff) which be found on www.mrc-bsu.cam.ac.uk/bugs/winbugs/
allows the use of complex functions via arbitrary remote14.shtml.

Vo lu me 3, September/Octo ber 2011 2011 Jo h n Wiley & So n s, In c. 395

 Overview wires.wiley.com/compstats

Note that WinBUGS was now stabilized in ver- algorithm implementation and output interpretation
sion 1.4.3 and it will not be developed any more. along with some toy examples, and a more detailed
Instead, the interest of the group is now turned on illustration that demonstrates the implementation of
the development of OpenBUGS (www.openbugs.info) Bayesian Lasso in WinBUGS. It can be used as a
which is an open source version of WinBUGS with brief introduction to WinBUGS for researchers with
additional features and contributions. This project or without statistical background.
started in 2004 when Andrew Thomas moved from WinBUGS is a useful computational tool that fits
London to Helsinki and now is stable and reliable complicated Bayesian models using MCMC methods.
package running under Windows, Linux, and MAC It is widely popular due to its numerous extensions and
operating systems. The possibility that researchers will
applications in various scientific fields. It is relatively
be able to contribute and improve an already popular
straightforward to use with syntax similar to the one
and successful software leaves high expectations for
in R and Splus. Alternatively, DOODLE interface can
the future.
be used to specify the structure of the model through
a graphical representation. The recent development of
CONCLUSION OpenBUGS (the open source version of the program)
This article summarizes the basic concepts required to creates high expectations for the future where any
perform Bayesian analysis using the WinBUGS. It pro- researcher will be able to contribute to the develop-
vides information on model specification and coding, ment and the improvement of this popular software.

REFERENCES
1. Lunn D, Spiegelhalter D, Thomas A, Best N. The bugs 7. Best N, Cowles MK, Vines K. CODA: Convergence
project: Evolution, critique and future directions. Stat Diagnostics and Output Analysis Software for Gibbs
Med 2009, 28:30493082. Sampling Output, Version 0.30. Cambridge: MRC Bio-
statistics Unit, Institute of Public Health; 1996.
2. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGS
User Manual, Version 1.4. MRC Biostatistics Unit, 8. Smith BJ. Bayesian Output Analysis Program (BOA),
Institute of Public Health and Department of Epidemi- Version 1.1.5 Users Manual, Technical Report. Depart-
ology and Public Health, Imperial College School of ment of Public Health, The University of Iowa,
Medicine, UK, 2003. Available at: http://www.mrc-bsu. 2005. Available at: http://www.mrc-bsu.cam.ac.uk/
cam.ac.uk/bugs/winbugs/contents.shtml. (Accessed bugs/winbugs/contents.shtml. (Accessed May 4, 2011).
May 4, 2011). 9. Tibshirani R. Regression shrinkage and selection via the
3. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGS lasso. J R Stat Soc [SerB] 1996, 58:267288.
Examples, vol 1. MRC Biostatistics Unit, Institute of 10. Lykou A, Ntzoufras I. On Bayesian Lasso Variable
Public Health and Department of Epidemiology and Selection and the specification of the shrinkage parame-
Public Health, Imperial College School of Medicine, UK, ter, Technical Report, Athens University of Economics
2003. Available at: http://www.mrc-bsu.cam.ac.uk/ and Business, 2011.
bugs/winbugs/contents.shtml. (Accessed May 4, 2011). 11. Kuo L, Mallick B. Variable selection for regression
4. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGS models. Sankhya, (B) 1998, 60:6581.
Examples, vol 2. MRC Biostatistics Unit, Institute of 12. Dellaportas P, Forster J, Ntzoufras I. On Bayesian model
Public Health and Department of Epidemiology and and variable selection using MCMC. Stat Comput 2002,
Public Health, Imperial College School of Medicine, UK, 12:2736.
2003. Available at: http://www.mrc-bsu.cam.ac.uk/ 13. Lunn D. WinBUGS Development Interface (WBDev).
bugs/winbugs/contents.shtml. (Accessed May 4, 2011). ISBA Bull 2003, 3:1011.
5. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGS 14. Lunn DJ, Whittaker JC, Best N. A Bayesian toolkit
Examples, vol 3. MRC Biostatistics Unit, Institute of for genetic association studies. Genet Epidemiol 2006,
Public Health and Department of Epidemiology and 30:231247.
Public Health, Imperial College School of Medicine, 15. Lunn DJ, Best N, Whittaker J. Generic reversible jump
UK, 2003. Available at: http://www.mrc-bsu.cam.ac. MCMC using graphical models. Stat Comput 2009,
uk/bugs. 19:395408.
6. Ntzoufras I. Bayesian Modeling Using WinBugs. New 16. Thompson JT, Palmer T, Moreno S. Bayesian analysis
York: John Wiley & Sons; 2009. in Stata using WinBUGS. Stata J 2006, 6:530549.

396 2011 Jo h n Wiley & So n s, In c. Vo lu me 3, September/Octo ber 2011