Modeling and Analysis of Stochastic Systems, Third Edition Vidyadhar G. Kulkarni
Modeling and Analysis of Stochastic Systems, Third Edition Vidyadhar G. Kulkarni
Modeling and Analysis of Stochastic Systems, Third Edition Vidyadhar G. Kulkarni
com
https://textbookfull.com/product/modeling-and-
analysis-of-stochastic-systems-third-edition-
vidyadhar-g-kulkarni/
https://textbookfull.com/product/modeling-and-analysis-of-dynamic-
systems-third-edition-esfandiari/
textbookfull.com
https://textbookfull.com/product/stochastic-modelling-for-systems-
biology-third-edition-darren-j-wilkinson/
textbookfull.com
https://textbookfull.com/product/biota-grow-2c-gather-2c-cook-loucas/
textbookfull.com
https://textbookfull.com/product/pharmaceutical-quality-by-design-a-
practical-approach-first-edition-gibson/
textbookfull.com
Trust me I m lying confessions of a media manipulator Ryan
Holiday
https://textbookfull.com/product/trust-me-i-m-lying-confessions-of-a-
media-manipulator-ryan-holiday/
textbookfull.com
https://textbookfull.com/product/mastering-bitcoin-programming-the-
open-blockchain-antonopoulos/
textbookfull.com
https://textbookfull.com/product/office-2013-elearning-kit-for-
dummies-1st-edition-faithe-wempen/
textbookfull.com
https://textbookfull.com/product/the-segmentation-of-europe-
convergence-or-divergence-between-core-and-periphery-1st-edition-mark-
baimbridge/
textbookfull.com
https://textbookfull.com/product/real-research-research-methods-
sociology-students-can-use-liahna-e-gordon/
textbookfull.com
All Rhodes Lead Here 1st Edition Mariana Zapata
https://textbookfull.com/product/all-rhodes-lead-here-1st-edition-
mariana-zapata/
textbookfull.com
Modeling and
Analysis of
Stochastic Systems
Third Edition
CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Series Editors
Francesca Dominici, Harvard School of Public Health, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Linear Algebra and Matrix Analysis for Modelling Binary Data, Second Edition
Statistics D. Collett
S. Banerjee and A. Roy Modelling Survival Data in Medical Research,
Mathematical Statistics: Basic Ideas and Third Edition
Selected Topics, Volume I, D. Collett
Second Edition Introduction to Statistical Methods for
P. J. Bickel and K. A. Doksum Clinical Trials
Mathematical Statistics: Basic Ideas and T.D. Cook and D.L. DeMets
Selected Topics, Volume II Applied Statistics: Principles and Examples
P. J. Bickel and K. A. Doksum D.R. Cox and E.J. Snell
Analysis of Categorical Data with R Multivariate Survival Analysis and Competing
C. R. Bilder and T. M. Loughin Risks
Statistical Methods for SPC and TQM M. Crowder
D. Bissell Statistical Analysis of Reliability Data
Introduction to Probability M.J. Crowder, A.C. Kimber,
J. K. Blitzstein and J. Hwang T.J. Sweeting, and R.L. Smith
Bayesian Methods for Data Analysis, An Introduction to Generalized
Third Edition Linear Models, Third Edition
B.P. Carlin and T.A. Louis A.J. Dobson and A.G. Barnett
Second Edition Nonlinear Time Series: Theory, Methods, and
R. Caulcutt Applications with R Examples
R. Douc, E. Moulines, and D.S. Stoffer
The Analysis of Time Series: An Introduction,
Sixth Edition Introduction to Optimization Methods and
C. Chatfield Their Applications in Statistics
B.S. Everitt
Introduction to Multivariate Analysis
C. Chatfield and A.J. Collins Extending the Linear Model with R:
Generalized Linear, Mixed Effects and
Problem Solving: A Statistician’s Guide,
Nonparametric Regression Models, Second
Second Edition
Edition
C. Chatfield
J.J. Faraway
Linear Models with R, Second Edition Nonparametric Methods in Statistics with SAS
J.J. Faraway Applications
A Course in Large Sample Theory O. Korosteleva
T.S. Ferguson Modeling and Analysis of Stochastic Systems,
Multivariate Statistics: A Practical Third Edition
Approach V.G. Kulkarni
B. Flury and H. Riedwyl Exercises and Solutions in Biostatistical Theory
Readings in Decision Analysis L.L. Kupper, B.H. Neelon, and S.M. O’Brien
S. French Exercises and Solutions in Statistical Theory
Discrete Data Analysis with R: Visualization L.L. Kupper, B.H. Neelon, and S.M. O’Brien
and Modeling Techniques for Categorical and Design and Analysis of Experiments with R
Count Data J. Lawson
M. Friendly and D. Meyer Design and Analysis of Experiments with SAS
Markov Chain Monte Carlo: J. Lawson
Stochastic Simulation for Bayesian Inference, A Course in Categorical Data Analysis
Second Edition T. Leonard
D. Gamerman and H.F. Lopes
Statistics for Accountants
Bayesian Data Analysis, Third Edition S. Letchford
A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson,
A. Vehtari, and D.B. Rubin Introduction to the Theory of Statistical
Inference
Multivariate Analysis of Variance and H. Liero and S. Zwanzig
Repeated Measures: A Practical Approach for
Behavioural Scientists Statistical Theory, Fourth Edition
D.J. Hand and C.C. Taylor B.W. Lindgren
Modeling and
Analysis of
Stochastic Systems
Third Edition
Vidyadhar G. Kulkarni
Department of Statistics and Operations Research
University of North Carolina at Chapel Hill, USA
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does not warrant the
accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB® software or related products
does not constitute endorsement or sponsorship by The MathWorks of a particular pedagogical approach or particular
use of the MATLAB® software.
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including pho-
tocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission
from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
my sons
Preface xix
1 Introduction 1
1.1 What in the World Is a Stochastic Process? 1
1.2 How to Characterize a Stochastic Process 5
1.3 What Do We Do with a Stochastic Process? 6
1.3.1 Characterization 7
1.3.2 Transient Behavior 7
1.3.3 First Passage Times 7
1.3.4 Limiting Distribution 8
1.3.5 Costs and Rewards 8
xi
xii CONTENTS
Epilogue 525
References 573
INDEX 579
Preface
Of course, if the results of Step 2 show that the model does not “fit” the real-life situa-
tion, then one needs to modify the model and repeat Steps 1 and 2 until a satisfactory
solution emerges. Then one proceeds to Step 3. As the title of the book suggests, we
emphasize the first two steps. The selection, the organization, and the treatment of
topics in this book are dictated by the emphasis on modeling and analysis.
Based on my teaching experience of over 25 years, I have come to the conclusion
that it is better (from the students’ points of view) to introduce Markov chains be-
fore renewal theory. This enables the students to start building interesting stochastic
models right away in diverse areas such as manufacturing, supply chains, genet-
ics, communications, biology, queueing, and inventory systems, etc. This gives them
a feel for the modeling aspect of the subject early in the course. Furthermore, the
analysis of Markov chain models uses tools from matrix algebra. The students feel
comfortable with these tools since they can use the matrix-oriented packages, such
as MATLAB R
, to do numerical experimentation. Nothing gives them better confi-
dence in the subject than seeing the analysis produce actual numbers that quantify
their intuition. We have also developed a collection of MATLAB R
-based programs
that can be downloaded from:
1. www.unc.edu/∼vkulkarn/Maxim/maxim.zip
2. www.unc.edu/∼vkulkarn/Maxim/maximgui.zip
The instructions for using them are included in the readme files in these two zip files.
After students have developed familiarity with Markov chains, they are ready for
renewal theory. They can now appreciate it because they now have a lot of renewal,
xix
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
xx PREFACE
renewal-reward, or regenerative processes models. Also, they are more ready to use
the tools of Laplace transforms.
I am aware that this sequence is contrary to the more prevalent approach that starts
with renewal theory. Although it is intellectually appealing to start with renewal the-
ory, I found that it confuses and frustrates students, and it does not give them a feel for
the modeling aspect of the subject early on. In this new edition, I have also changed
the sequence of topics within Markov chains; I now cover the first passage times be-
fore the limiting behavior. This seems more natural since the concepts of transience
and recurrence depend upon the first passage times.
The emphasis on the analysis of the stochastic models requires careful develop-
ment of the major useful classes of stochastic processes: discrete and continuous time
Markov chains, renewal processes, regenerative processes, and Markov regenerative
processes. In the new edition, I have included a chapter on diffusion processes. In or-
der to keep the length of the book under control, some topics from the earlier edition
have been deleted: discussion of numerical methods, stochastic ordering, and some
details from the Markov renewal theory. We follow a common plan of study for each
class: characterization, transient analysis, first passage times, limiting behavior, and
cost/reward models. The main aim of the theory is to enable the students to “solve”
or “analyze” the stochastic models, to give them general tools to do this, rather than
show special tricks that work in specific problems.
The third aspect, the implementation, involves actually using the results of Steps
1 and 2 to manage the “real-life” situation that we are interested in managing. This
requires knowledge of statistics (for estimating the parameters of the model) and
organizational science (how to persuade the members of an organization to follow
the new solution, and how to set up an organizational structure to facilitate it), and
hence is beyond the scope of this book, although, admittedly, it is a very important
part of the process.
The book is designed for a two-course sequence in stochastic models. The first
six chapters can form the first course, and the last four chapters, the second course.
The book assumes that the students have had a course in probability theory (measure
theoretic probability is not needed), advanced calculus (familiarity with differential
and difference equations, transforms, etc.), and matrix algebra, and a general level
of mathematical maturity. The appendix contains a brief review of relevant topics. In
the second edition, I have removed the appendix devoted to stochastic ordering, since
the corresponding material is deleted from the chapters on discrete and continuous
time Markov chains. I have added two appendices: one collects relevant results from
analysis, and the other from differential and difference equations. I find that these
results are used often in the text, and hence it is useful to have them readily accessible.
The book uses a large number of examples to illustrate the concepts as well as
computational tools and typical applications. Each chapter also has a large number
of exercises collected at the end. The best way to learn the material of this course
is by doing the exercises. Where applicable, the exercises have been separated into
three classes: modeling, computational, and conceptual. Modeling exercises do not
PREFACE xxi
involve analysis, but may involve computations to derive the parameters of the prob-
lem. A computational exercise may ask for a numerical or algebraic answer. Some
computational exercises may involve model building as well as analysis. A concep-
tual exercise generally involves proving some theorem, or fine tuning the understand-
ing of some concepts introduced in the chapter, or it may introduce new concepts.
Computational exercises are not necessarily easy, and conceptual exercises are not
necessarily hard. I have deleted many exercises from the earlier edition, especially
those that I found I never assigned in my classes. Many new exercises have been
added. I found it useful to assign a model building exercise and then the correspond-
ing analysis exercise. The students should be encouraged to use computers to obtain
the solutions numerically.
It is my belief that a student, after mastering the material in this book, will be well
equipped to build and analyze useful stochastic models of situations that he or she
will face in his or her area of interest. It is my fond hope that the students will see
a stochastic model lurking in every corner of their world as a result of studying this
book.
What’s new in the third edition?
I have added several new applications in the third edition, for example, Google
search Algorithm in discrete time Markov chains, several health care and finance
related applications in the continuous time Markov chains, etc. I have also added
over fifty new exercises throughout the book. These new exercises were developed
over the last ten years for my own exams in the course and the qualifying exams for
our PhD candidates. I have heeded the request from the instructors using this as a
textbook not to change the exercise numbers, so the new exercises are added at the
end in appropriate sections. However this means that the exercise sequence no longer
follows the sequence in which topics are developed in the chapter. To make space for
these additions, I have deleted some material from the second edition: most notably
the section on extended key Markov renewal theorem and the related conceptual
exercises have been deleted. The material in Chapter 10 on diffusion processes has
been rewritten in several places so as to make it more precise and clearer. Many
graduate students have helped me find and correct the embarrassingly many typos
that were present in the second edition. In particular, Jie Huang has helped proofread
the third edition especially carefully, and I thank her for her help. However, there are
bound to be a few new typos in the third edition. I will appreciate it if the readers are
kind enough to send me an email when they find any.
Vidyadhar Kulkarni
vkulkarn@email.unc.edu
Department of Statistics and Operations Research
University of North Carolina
Chapel Hill, NC
MATLAB R
is a registered trademark of The MathWorks, Inc. For product informa-
tion please contact
Introduction
The discipline of operations research was born out of the need to solve military
problems during World War II. In one story, the air force was using the bullet holes
on the airplanes used in combat duty to decide where to put extra armor plating. They
thought they were approaching the problem in a scientific way until someone pointed
out that they were collecting the bullet hole data from the planes that returned safely
from their sorties.
Consider a system that evolves randomly in time, for example, the stock market
index, the inventory in a warehouse, the queue of customers at a service station,
water level in a reservoir, the state of a machines in a factory, etc.
Suppose we observe this system at discrete time points n = 0, 1, 2, · · ·, say, every
hour, every day, every week, etc. Let Xn be the state of the system at time n. For
example, Xn can be the Dow-Jones index at the end of the n-th working day; the
number of unsold cars on a dealer’s lot at the beginning of day n; the intensity of the
n-th earthquake (measured on the Richter scale) to hit the continental United States
in this century; or the number of robberies in a city on day n, to name a few. We say
that {Xn , n ≥ 0} is a discrete-time stochastic process describing the system.
If the system is observed continuously in time, with X(t) being its state at time
t, then it is described by a continuous time stochastic process {X(t), t ≥ 0}. For
example, X(t) may represent the number of failed machines in a machine shop at
time t, the position of a hurricane at time t, or the amount of money in a bank account
at time t, etc.
More formally, a stochastic process is a collection of random variables
{X(τ ), τ ∈ T }, indexed by the parameter τ taking values in the parameter set T .
The random variables take values in the set S, called the state-space of the stochastic
process. In many applications the parameter τ represents time, but it can represent
any index. Throughout this book we shall encounter two cases:
1
2 INTRODUCTION
1. T = {0, 1, 2, · · ·}. In this case we write {Xn , n ≥ 0} instead of {X(τ ), τ ∈ T }.
2. T = [0, ∞). In this case we write {X(t), t ≥ 0} instead of {X(τ ), τ ∈ T }.
t
(a) Continuous-time, discrete state-space
X(t)
t
(b) Continuous-time, continuous state-space
Xn
× × × ×
× × ×
× × ×
× × ×
n
(c) Discrete-time, discrete state-space
We have had a few other collegiate and university locals but they
did not prove very long-lived, and it was very difficult for us to get
detailed reasons for their decline. I presume fear would account for
most of them.
CHAPTER XC
THE PROFESSORS’ STRIKE
The final purpose of this book, you will now realize, is to bring
about a strike of college professors. The next question to be
considered is, what are the principles upon which this strike shall be
based?
First and foremost, the question of tenure; which is exactly the
same thing as the claim of the worker to security in his job. The
college professor must not forfeit his standing except for cause, and
upon due and reasonable notice. He must have the right which every
criminal possesses, of knowing what are the charges against him,
and of having a hearing in which he is confronted by his accusers,
and given the right to cross-question them, and to answer their
charges and prove them false if he can. The decision in his case
must rest, not with his masters and exploiters, but with his fellow-
workers; in other words, the ancient right embodied in Magna Carta,
to be tried by a jury of his peers. These rights are elemental; there
can be no freedom, no dignity or self-respect for any man who does
not possess them. They are possessed by scholars in all other
civilized countries; it is only in our sweet land of liberty that scholars
are slaves. Says James McKeen Cattell:
Abelard, 454
Abortions, 381
“Abrams case,” 75
“Acres of Diamonds,” 332
Advertising, 315
Allegheny, 347
Allen, F. J., 89
Alumni, LXXIII
Amal. Clothing Workers, 452
“A Man’s World,” 295
American, 349
Amer. Ass’n of University Profs., 181, 186, 192, 195, 346–7, 354,
375, 409, 455
Amer. Book Co., 289
Amer. Civil Lib. Union, 475
Amer. Fed. of Teachers, 459
Amherst, 432
Ammons, 193