REVIEWING BASIC SCIENCES
Markov Models in health care
Modelos de Markov aplicados a saúde
Renato Cesar Sato1, Désirée Moraes Zouain2
ABSTRACT
Markov Chains provide support for problems involving decision
on uncertainties through a continuous period of time. The greater
availability and access to processing power through computers
allow that these models can be used more often to represent clinical
structures. Markov models consider the patients in a discrete state
of health, and the events represent the transition from one state
to another. The possibility of modeling repetitive events and time
dependence of probabilities and utilities associated permits a more
accurate representation of the evaluated clinical structure. These
templates can be used for economic evaluation in health care taking
into account the evaluation of costs and clinical outcomes, especially
for evaluation of chronic diseases. This article provides a review of
the use of modeling within the clinical context and the advantages of
the possibility of including time for this type of study.
Keywords: Health economics; Markov chains; Models, economic
RESUMO
Os modelos de Markov prestam apoio aos problemas de decisão
envolvendo incertezas em um período contínuo de tempo. A maior
disponibilidade e o maior acesso no poder de processamento por meio
dos computadores permite que esses modelos possam ser utilizados
mais frequentemente para representar estruturas clínicas. Os
modelos de Markov consideram os pacientes em um estado discreto
de saúde, e os eventos representam a transição de um estado para
outro. A possibilidade de modelar eventos repetitivos e a dependência
temporal das probabilidades e utilidades associadas permitem uma
representação mais precisa da estrutura clínica avaliada. Esses
modelos podem ser utilizados para avaliações econômicas em saúde
levando em consideração a avaliação dos custos e desfechos clínicos
(outcomes), especialmente para a avaliação de doenças crônicas.
Este artigo oferece uma revisão do uso dessa modelagem dentro do
contexto clínico e as vantagens da possibilidade da inclusão temporal
para esse tipo de estudo.
Descritores: Economia da saúde; Cadeias de Markov; Modelos
econômicos
INTRODUCTION
Economic decision models have been increasingly used
to assess health interventions(1,2). Advances in this field
are mainly due to enhanced processing capacity of
computers, availability of specific software to perform
these tasks, and sophisticated mathematical techniques,
which have become more popular.
Due to the reasons pointed out above, more
investigators adopted the Markov models, which
historically had already been used in epidemiological
and clinical evaluations (3). In health economics, the
strength of Markov models is that they take into
consideration the use of resources and the outcomes.
In this review the authors discuss the use of Markov
models for economic evaluations of the health sector.
This work introduces a structure to evaluate health
programs, the use of the Markov model, its variables
and structure of analysis.
Authors agree that economic evaluations in health
care should be carried out to deal with the introduction
of new technologies, based on an analytic decision
model under conditions of uncertainty(4-8). This model
follows the following decision making process:
1. structure: must adequately reflect the possibility of
prognosis that individuals may undergo, and the
impact that treatment and health programs have on
said prognosis. In this situation, the individuals are
usually patients with a specific health condition, but
may be healthy or asymptomatic, as in prevention
campaigns;
2. evidence: provides an analytical structure in which
relevant evidence for the study may be defined. This
could be obtained through the model and through
the entry parameters;
3. evaluation: provides a mean of translating relevant
evidence into cost estimates and comparison of the
1
PhD Student at the Instituto de Pesquisas Energéticas e Nucleares (IPEN) of Universidade de São Paulo – USP, São Paulo (SP), Brazil.
2
PhD in Nuclear Technology and Management of Innovations and Technology; Lecturer at the Graduate Program of Instituto de Pesquisas Energéticas e Nucleares (IPEN) of Universidade de São Paulo –
USP; Coordinator of the Center of Management and Technological Policy of Universidade de São Paulo – USP, São Paulo (SP), Brazil.
Received on Dec 9, 2009 – Accepted on Jun 28, 2010
Corresponding author: Renato Cesar Sato – Avenida Martin Luther King, 2.386 – Vila São Francisco – CEP 05352-020 – São Paulo (SP), Brasil – Tel.: (11) 2151-1233 – e-mail: rcsato@ipen.br
einstein. 2010; 8(3 Pt 1):376-9
Markov Models in health care
impact of the options under comparison. The main
types of study are cost-effectiveness, cost-benefit
and cost-utility. The best option must be treated
based on the evidence available;
4. uncertainty and variability: enables an evaluation
of numerous types of uncertainty, including those
related to the model and the entry parameters. The
models must also provide flexibility to characterize
heterogeneity through several subgroups of
individuals;
5. future research: through the evaluation of
uncertainties, it is possible to identify priorities for
future research, which will produce evidence to reevaluate the issue in the future.
Following this decision-making process, the
economic evaluation seeks information regarding the
process of appropriate data measurement and adequate
information about the distribution of resources(8)
according to the uncertainty at hand. Chart 1 contains
a brief description of the types of possible uncertainties
and the possible approaches to deal with them.
377
expected costs and outcomes. How fast patients move
from one state of the model to the next is determined by
the probability of the transitions. Thus, by determining
the use of resources and outcomes in health, it is possible
to evaluate these factors associated to the disease and
the intervention that is performed.
The first stage in the construction of a Markov
model is defining the different states of the disease.
These states must represent the important clinical and
economic effects of the disease, and said effects should
be included in the model. One important consideration
is that these stages of disease are mutually exclusive,
because the patient cannot be in more than one state of
the disease at the same time.
With the development of chronic diseases, such as
hypertension and diabetes in developing countries(912)
, Markov models became important tools for
planning health care programs. Figure 1 is a graphic
representation of chronic disease that may be introduced
in an economic evaluation model.
Chart 1. Types of uncertainties and possible approaches
Type of uncertainty
Methodological
Sample variation
Extrapolation
Transferability
Possible approach
Reference case/sensitivity analysis
Statistical analysis
Modeling methods
Sensitivity analysis
In the present study, the authors addressed the
problem of uncertainty through extrapolation using
Markov models.
TIME IN MARKOV MODELS
The most important difference between the Markov
models and other models of economic evaluation in
health science is the state of a patient during a specific
moment in time. The factor “time” is explicitly associated
with the probability of a patient taking certain states in
a series of discrete periods of time. In Markov models,
these periods are called “cycles”. In other words, a
disease is divided in distinct cycles, and probabilities are
attributed to the transition between these states. The
duration of these cycles depends on the disease and on
the interventions that are being evaluated, and may be
monthly or annual cycles, for example. From the point
of view of economic evaluation, a cost is associated with
each cycle, except in the case of cost-utility studies, in
which the value represents the utility associated to each
cycle. The average amount of time that a patient spends
on the various states of the model is then weighted
by cost or utility, which will be used to calculate the
Figure 1. Stages of the disease progression until death(1)
The first state is defined as asymptomatic and
indicates that the patient suffers from the disease, but is
not experiencing its consequences and the risk of death
is not higher than in someone who does not have the
disease. From this state of the disease on, the patient
may move towards the stage of “death”, based on the
probability of transition or progression of the disease.
In disease progression, the patient starts to experience
einstein. 2010; 8(3 Pt 1):376-9
378
Sato RC, Zouain DM
the health impairment with an increased risk of death
caused by the direct result of the disease on all other
causes of mortality.
Absorbing state is a state in the model from which
it is technically impossible to move out, and an example
is death. The backward arrows indicate the possibility
of the patient remaining in this state or, according to
the model, it is possible to include improvements in
the clinical conditions of the patient, as in the case of
disease remission.
The probabilities of transition are considered in each
cycle of the model, and may be represented in a matrix
of the type “n x n”. The sum of probabilities of transition
of each cycle must be equal to 1 (one), because there is
only one state in each discrete moment of time. Thus, the
probability of remaining in the same state is given by the
value 1 (one) minus the probability of transition.
For the purpose of illustration, table 1 presents a
matrix of probabilities in a monotherapy state.
Table 1. Probabilities of transition in a monotherapy case(13)
Transition from
State A
State B
State C
State D
Transition to
State A
0.721
0
0
0
State B
0.202
0.581
0
0
State C
0.067
0.407
0.75
0
State D
0.01
0.012
0.25
1
One important observation is the possible confusion
in the use of the terms “rate” and “probability”. Rate
represents the transition in any given point in time,
whereas probability is the proportion of the population
at risk in a specific period of time(1,14). Therefore,
probabilities available in the literature may not reflect
the same period of time in the Markov cycle of the
model in use. Data retrieved from the academic
literature is usually expressed in rates that may vary
from 0 to indefinite (example: a mortality rate of 2% a
year for disease X), whereas probabilities vary from 0 to
1 during a specific period of time.
The issue of probabilities may be avoided if:
the cycle in years produces the life expectancy in years.
In economic evaluations, it is common to observe this
estimate in terms of quality-adjusted life years (QALY).
The main advantage resides in averaging the extension
of time and a specific state of health represented by
the quality of life during this state of health. This
makes QALY appropriate for use in Markov models.
The use of cost follows the same method, attributing
values spent during each cycle, and the cost is obtained
through the sum of cycles. One consideration is the
possibility to attribute cost not only to the states, but
also to the transitions, that can represent punctual
treatments.
Markov models are widely used in science, including
areas such as Biology, Mathematics, Social Science,
Music, internet, Chemistry and Physics.
The strengths of this type of model are the constant
transition probabilities that may be solved by matrix
algebra, considering that the matrix of transition
contemplates the time spent in each state and the
expected value of each outcome in a precise manner.
However, the weakness of this type of model is also the
statistic probability. As mentioned above, these methods
became more popular due to the higher processing
capacity of computers, which are able to overcome the
statistical limit of matrix analysis.
Below is a review of the two main types of Markov
models used to assess health care programs – cohort
simulation and individual simulation.
The cohort simulation offers a direct solution. The
cohort initiates at the moment 0 in an initial state of
disease; in our example of chronic disease, it corresponds
to the asymptomatic condition of the disease. For each
cycle of the model the transition probabilities are applied
and the distribution of patients in each state is adjusted.
The execution of several cycles determines the profile
of how many patients exist in each state of the model
throughout time. Table 2 presents a hypothetical cohort
model with 1,000 patients.
Table 2. Simulation by cohort for the example model(1)
P(t)=1 e(rt)
where time is expressed as “t” and rate as “r”(15).
The attribution of weights to the model is needed
in order to estimate costs and outcomes. In the case
of life expectancy, the weight 1 (one) is attributed to
each state of the model in which the patient is “alive”,
whereas 0 (zero) is attributed to the state of “dead”.
The average life expectancy in terms of size of the
model cycle is obtained through performing the model
over a large number of cycles and adding up the values
throughout these cycles. This multiplied by the size of
einstein. 2010; 8(3 Pt 1):376-9
Cycle
0
1
2
3
4
5
6
7
8
9
10
State of the disease
Asymptomatic
1,000
976
943
902
854
799
740
678
614
551
488
Progressive
0
10
28
52
79
109
139
168
195
218
237
Death
0
14
29
46
67
92
121
154
191
231
275
Total
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
Markov Models in health care
In the Monte Carlo simulations, instead of initiating
a cohort of the model, a great number of patients are
followed up individually. The main difference is that
despite the fact that individual patients are subject to
the same transition probabilities, in the Monte Carlo
model, they may or may not “move” between the stages
of each cycle. Thus, the path followed by different
patients will vary, due to the random variability; whereas
cost and outcomes are produced according to the path
followed in the model.
Despite the growth of economic analysis in clinical
testing, it is clear that the main economic advantages may
be translated in better distribution of scarce resources.
Economic evaluations are a preliminary stage in
clinical studies and Markov models are adequate to
evaluate the progression of disease throughout time.
Another important advantage is the ability to deal with
costs and effects simultaneously. Similar to all models,
Markov models have limitations that must be overcome
as models become more sophisticated, especially when
dealing with time-dependent probabilities of transitions
and different states of disease. Another inherent
difficulty of this type of model is the greater complexity
when compared to the simpler decision-making trees,
and the lack of “memory”. This is due to the assumption
of Markov regarding the probability of moving between
states of health disregarding the experiences of previous
cycles(16). This can be reduced with the “tunnel states”
that enable integration of health experiences from
the previous cycles(16,17). The states of the cycles can
be accessed only in a pre-determined sequence, an
analogy to passing through a tunnel. The purpose of
this approach is to offer a temporary adjustment in the
probability transitions that last longer than one cycle.
In health sciences, Markov models are widely used
as analytical tools to assess diseases from an economic
point of view. According to this technique, a patient
may be assessed in a finite number of discrete states
of health, in which the important clinical events are
modeled as transitions from one state to another. The
studies involving the Markov chains may be presented
simulations, such as cohort; that is, a trial with multiple
subjects, or through a Monte Carlo simulation, involving
multiple trials and one subject for each.
These studies are presented as cycle trees that
combine the structure of decision-making with the
379
Markov processes. This enables the consideration of
clinical problems with continuous risks throughout
time within a model. The diffusion of these techniques
may contribute in clinical assessments of the current
moment, which involves increases in cost and in the
prevalence of chronic diseases.
REFERENCES
1. Briggs A, Sculpher M. An introduction to Markov modelling for economic
evaluation. Pharmacoeconomics. 1998;13(4):397-409.
2. Buxton MJ, Drummond MF, Van Hout BA, Prince RL, Sheldon TA, Szucs T, et
al. Modelling in economic evaluation: an unavoidable fact of life. Health Econ.
1997;6(3):217-27.
3. Sonnenberg FA, Beck JR. Markov models in medical decision making: a
practical guide. Med Decis Making. 1993;13(4):322-38.
4. Elsinga E, Rutten FF. Economic evaluation in support of national health policy:
the case of The Netherlands. Soc Sci Med. 1997;45(4):605-20.
5. Sculpher MJ, Claxton K, Drummond M, McCabe C. Whither trial-based
economic evaluation for health care decision making? Health Econ.
2006;15(7):677-87.
6. Buxton MJ. Economic evaluation and decision making in the UK.
Pharmacoeconomics. 2006;24(11):1133-42.
7. Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health
technology assessments: a systematic review of current practical approaches.
Int J Technol Assess Health Care. 2007;23(3):310-5.
8. Drummond MF, Sculpher MJ, Torrance GW, OBrien BJ, Stoddart GL. Methods
for the economic evaluation of health care programmes. 3rd ed. Oxford:
University Press; 2005.
9. Zimmet P, Alberti KG, Shaw J. Global and societal implications of the diabetes
epidemic. Nature. 2001;414(6865):782-7.
10. Abegunde DO, Mathers CD, Adam T, Ortegon M, Strong K. The burden and
costs of chronic diseases in low-income and middle-income countries. Lancet.
2007;370(9603):1929-38.
11. Yach D, Leeder SR, Bell J, Kistnasamy B. Global chronic diseases. Science.
2005;307(5708):317.
12. Adeyi O, Smith O, Robles S. Public policy and the challenge of chronic
noncommunicable diseases. Washington: The World Bank; 2007.
13. Chancellor JV, Hill AM, Sabin CA, Simpson KN, Youle M. Modelling the cost
effectiveness of lamivudine/zidovudine combination therapy in HIV infection.
Pharmacoeconomics. 1997;12(1):54-66.
14. Miller DK, Homan SM. Determining transition probabilities: confusion and
suggestions. Med Decis Making. 1994;14(1):52-8.
15. Beck JR, Pauker SG. The Markov process in medical prognosis. Med Decis
Making. 1983;3(4):419-458.
16. Rascati, K. Essentials of pharmacoeconomics. Philadelphia: Wolters Kluwer;
2009.
17. Hawkins N, Sculpher M, Epstein D. Cost-effectiveness analysis of treatments
for chronic disease: using R to incorporate time dependency of treatment
response. Med Decis Making. 2005;25(5):511-9.
einstein. 2010; 8(3 Pt 1):376-9