2004 IEEE Intemational Conference on Electric Utility Deregulation, Restructuring and Power Technologies (DRPT2004) April 2004 Hong

Kong

Basic Analysis of the Pricing Processes in Modeled

Electricity Markets with Multi-Agent Simulation
Yasumasa Fujii, Member, IEEE, Tomoaki Okamura, Non Member
Kenji Inagaki, Non Member, and Kenji Yamaji, Non Member

Abstract- Many electric utilities world-wide have been equilibrium solutions in consideration of power transmission
forced to change their ways of doing business, from constrains [ I ] has been done, but it is pointed out that to explore
vertically integrated systems to open market ones. In Japan, the global optimal solution is difficult because there are so many
\
we are facing urgent issues about how to design the local optimal solutions.
structures of power market. In order to cope with the issue, There is another method based on reinforcement learning. In
many studies have been made with market models. However, this method, a market is simulated with artificial multi-agents
there are many kinds of modeling methods, and no developed on computers. Every agent in the model explores its
particular method has been established so far. Each has own action automatically. There have been some studies in
drawbacks and advantages about validity and versatility. consideration of a day's load fluctuation with multi-agent
This paper presents the method to use a multi-agent model, systems [2]. However, the effects of market designs on pricing
which learns a bidding strategy autonomously through trial- processes have not been studied sufficiently yet.
and-error search action, as useful tools of numerical analysis We proposed utilization of multi-agent models based on the
of the price formation process of an electric power market. reinforcement learning theory to work out numerically Nash-
The model enables us to analyze more general electricity
equilibrium. By using this model, we can derive the optimal
markets, which have several different types of power plants
Nash-equilibrium solution because agent explores the optimal
with unit commitment costs, seasonal and hourly demand
solution as they learn their respective strategies by trial and error.
fluctuation, real-time regulation market and operating
reserve market. This model figures out the issues that may In this paper, we mainly analyze relations between market
be appeared when we will design markets and the way of designs and bidding strategies of power plant agents by using
designing operating reserve market. our agent-based model, and indicate the usefulness of the model.

Index Terms--Game theory, Intelligent agent interaction, 11. MARKET MODELING

Learning systems, Power system economics, Power system
planning A. Market Modeling
In this model, an entity that determines its policy to acquire
1. INTRODUCTION profits by selling electricity is called a supplier agent, and an
entity that acquires utility by buying electricity is called a
P ower supply systems have been changing to systems based
on market mechanism as deregulations of electric industries
have been carried out in many parts of the world since 1970s.
consumer agent. We introduce the set of the supplier agent
G={A,,: i=l, ..., n } and that of the consumer agents D={A4:
Conventional pricing process based on full-cost principle, j=l, ..., m}.The supplier agents and the consumer agents have
however, cannot be applied, because electricity prices in their respective cost and utility functions. C,(q) represents the
deregulated markets are determined by multiple decision- cost function of i-th supplier agent, and U,(q) represents the
making entities that have their own objectives. Therefore, it is so utility function of j-th consumer agent. We assume quadratic
important to establish the methods to analyze the pricing form for these functions as below.
processes in a deregulated electricity market.
There is a method based on game theory to design the
market. The purpose of the method is to work out equilibrium
solution to maximize profit of every market participant. If we Where agl,&, ys. a+ & and yd are coefficients that have
can work out one of Nash equilibrium solutions concretely for
the following sign conditions of cx,,>O, >O, ad<Oand &>O.
market participants' strategies, we can discuss mathematically
QS and Q4 are the upper limits of q with respect to agents Azt
market characteristics that affect pricing process. It is, however,
and Ad, respectively.
extremely difficult to calculate analytically Nash equilibrium
Marginal cost and marginal utility are represented liner
solution for general problems. A study to work out Nash function as below.

Yasumasa Fujii is with the Dcpartmcnt of Electrical Engineering,

M c (4) = 2a,q + PgJ (0 5 q 5 QgJ (3)
Graduate School of Engineering, University of Tokyo, 113-8656 Japan
(e-mail: yas@ce.t.u-tokyo.ac.jp).
Tomoaki Okamura is with the Department of Electrical Engineering, In this model, supplier agent Agrand consumer agent As are
Graduate School of Engineering, University of Tokyo, 113-8656 Japan assumed that they decide their bidding curves by adjusting the
(e-mail: okamura@yamaji.t.u-tokyo.ac.jp). biases as, and adiof their respective marginal cost and marginal
Kcnji Inagaki is with the Department of Advanced Energy, Graduate utility curves, as shown in Fig 1 [ 3 ] .
School of Frontier Scicnce, University of Tokyo, 113-8656 Japan (e- The supply curve and demand curve of the whole market can
mail: inagaki@yamaji.t.u-tokyo.ac.jp).
Kenji Yamaji is with the Department of Advanced Energy, Graduate be drawn by summing up the bid curves of each agent. The
School of Frontier Scicnce, University of Tokyo, 113-8656 Japan (e- market price P' and each agent's quantity of transaction qS and
mail : yamaj iOyamaj i .t.u-tokyo.ac .jp) . qdi are cleared based on these bids.

0-7803-8237-4/04/$17.0002004IEEE
257
2004 IEEE International Conference on Electric Utility Deregulation, Restructuring and Power Technologies (DRPT2004) April 2004 Hong Kong

case of reference point (100,00OMWh, 5yedkWh). Fig.4 (b) included in action-value function is 144 (seasons: 3, hours: 24,
indicates daily load curves for 3 seasons (summer, winter, markets: 2). For one episode, agents acquire their own rewards
others) at electricity price of 5yenikVJh. through two markets' clearance transacted every hour. We use
Power exchange market trade is typically forward trade (e.g. Temporal Difference (TD) method ( ~ 0to)get feedback agent's
day-ahead or hour-ahead), and an IS0 (Independent System reward into agent's action-value function. In a certain seasons (s
Operator) has to regulate energy balance of supply and demand €3) and hour h (hE24), if the reward of one day becomes R
before electric energy is supplied to customers. To take account when it acts on an action pair of (apx,a,) in the two market mpx
of the uncertainty of real time demand of electricity, we assume and m,, its action-value function is updated as below.

1
stochastic electricity demands of which the average magnitude is r )Q k ( ~ , h , m ~ ~ , a , , )
Qkii(~,kmpn,ap=
5 % of the reference demand of each corresponding time. The
probability distribution of the stochastic demands is assumed to
+ &- Qk apxI]
(s, h,mpx3
(9)
be logarithmic normal distribution. Q,,, (s, h,m, ,a,) = Q, (s,h,m, A, 1
+ a [ R-Q,(s,h,m,,a,)I
B. Modeling of generator agents
This sequential flow of calculation is shown in Fig7, in this
We assume five types of plants (Hydro, Nuclear, Coal, LNG, case, however, we do not consider consumer agents' bid
and Oil), and each agent has one of these types of plant. Power strategies because we found that consumer agents comply with
plant mixes and marginal costs are shown in Fig.5, and facility assumed demand curve when there are many consumer agents in
costs of each type of plant are shown in Table I. Load following our previous study [ 5 ] .
constraints of power plants are not considered in this paper.

PX Market Mechanism
Coal Energ, / Capacity
.*1

(a) Power plant mixes (b) Marginal costs

Fig.5 Power Plant Mixes and Marginal Cost Characteristics in
the Electricity Market

TABLE I
I
FACILITYCOSTS OF EACHTYPEOF PLANT
I Hvdro I Nuclear I Coal I LNG I Oil
Unit Facility Cost
404 377 304 214 206
Life time 40 50 50 50 50
[year]
Annual Expense 5.5% 7.9% 7.7% 7.7% 7.7%
Rate

C. Modeling of electricity market

This study takes account of both power exchange market and Fig.7 Algorithm of Electricity Market Model
real-time market. The real-time market is transacted just before
supplying electric power. First, in power exchange market, Iv. RESULTS OF SIMULATION
supplier and consumer agents bid based on their marginal cost
and their marginal utility as shown in Fig.6 (a). Next, supplier A. Market price f o r share of each plant
agents bid again and an I S 0 makes the determination on the First in this section, we evaluate the effects of shares of each
basis of their bids as shown in Fig.6 (b). Power flows based on type of plant on electricity market by using this model. The total
transmission constraints are not considered in this paper. capacity of the market is assumed to be 120% of the average of
peak demand including stochastic real time demand.
P
Demand C w e
I
This paper considers 3 cases (base case, hydro monopoly
case, and oil monopoly case). Base case is the case where there
are 50 suppliers, which every ten agents has same types of
power plants. Hydro monopoly case is the case where there
exists only one agent for hydro power plant in the market. Oil
monopoly case is the case where there exists only one agent for
QPX' Q QKRT. Q oil-fired power plant in the market.
(a) Power Exchange market (b) Real-time market Fig.8 shows quantities of transactions and power plant
Fig.6 Market Processes operation for a certain week on summer in the base case. In the
operating reserve market, quantity of transaction is assured
D. Application of reinforcement learning theory sufficiently in a real-time market for load fluctuations as shown
As we said above, we are taking account of two markets in in Fig.8 (a). Hydro power and nuclear power plants of which
this model. Therefore, each agent's number of states that are

258
2004 IEEE Intemational Conference on Electric Utility Deregulation, Restructuring and Power Technologies (DRPT2004) April 2004 Hong Kong

marginal costs are low are utilized as base loaf power plants, and institutions on electricity market are simulated here.
oil-fired power plant of which the marginal cost is rather high is I ) VOLL (Value Of Lost Load) pricing
utilized just as peak load power plants as shown in Fig.8 (b). Electricity price will rise to infinity when suppliers cannot
Price duration curves of one year are shown in Fig.9. be satisfied customers’ demand sufficiently because price
elasticity of customers is zero in the real-time market. VOLL
lHydro UNuclear =Coal pricing is institution based on marginal value pricing in electric
OLNG =Oil -Caoaotv
lSWO0, I
power shortage. This process in VOLL pricing is shown in
FiglO. In this model, price cap: 250[yen/kWh] is set up as taking
consideration into modeling of a marginal cost curve.

‘t I Real Demand

0 24 48 72 88
Hour [ h 1
120 I44
P,,,=VOLL
I ---

(a) Quantity of transaction (b) Power plant operation

Fig.8 Market Quantity and Plant Operation
60 I I Fig. 10 Real-time Market Process in VOLL Pricing
Power Exchange Market
2) Operating reserve pricing
In this institution demand-side supply entities are obliged to
reserve capacity of supply more than a certain level to assure
supply reliabilities. The supply entities that do not have their
own sufficient capacities have to buy remaining capacities in a
1 8760 market. In this model, we assume that supplier agents procure
Duration Hours [h] the capacities in the real-time market as well, this market is
300
I modeled so that the market consists of an real time imbalance
i250 . Real Time Market energy market and an operating reserve market, as shown in
$200 Fig. 1 1. In this model, the price cap: P (yen/kWh) is set up in the
only case that supply entities cannot satisfy the given level of
operating reserve rate: Ph.

t Real Demand p t Operating Reserve Pt Real Demand

1 8760
Ouration Hours [h]

Fig.9 Price Duration Curves _________-_______

S pply Curv
It is clear that the price duration curves in the two markets
are the almost same for both the base case and the hydro
Operating Reserve
monopoly case as shown in Fig.9. For the oil monopoly case, on Fig. 11 Real-time Imbalance Energy and Operating Reserve
the other hand, market prices become higher in the two markets, Market Process
especially it is significant in the real-time market. This is
because that the real time market is almost same as the C. Effects on marketfor market structure
monopoly market of oil-fired power plant, and there is almost no We assume the case that the total market capacity equals to
incentive to keep the price low. 105% of average peak load as the case where electric supply and
For the hydro monopoly case and the oil monopoly case, demand balance is stringent. Loads of consumers are curtailed as
HHI index defined as (IO) is about 0.04, because the market having regular probabilities on account of irregular demand at
shares of monopolistic agents are 20%. the real-time market. Load duration curves of one year in the
two cases are shown in Fig.12. In these cases, the price cap
HHI = Is:,
r=l
equals to IOOyenikWh in the VOLL pricing, level of operating
reserve rate equals to 10% in the operating reserve. Load
Where sgl is market share of i-th power plant.
durations are almost same in each structure, and it is
The market of the oil monopoly case cannot be a
understandable that the loads are curtailed for a few hours in a
monopolistic market as a whole. However, the expansion of
year.
market share of a power plant agent for peak load will provide it
Next, price duration curve in real-time market are shown in
with a chance to obtain decisive market power. Therefore, we
Fig.13. Price duration in power exchange market is cut out
should take account of the market share of power plant for peak
because we could not find out a noticeable difference between
load in designing an electricity market.
two institutions. In the real-time market, however, price duration
B. Design of pricing and electric supply reliability curve of the operating reserve case equals to price cap of 100
We assume the case where electric supply and demand y e n k w h for a longer time when operating reserve rate falls
balance is stringent, and then pick up VOLL pricing and below 90%, and price duration curve of the VOLL pricing case
operating reserve as institutions. The effects of the two equals to price cap of 250 y e n k w h for a few hours per year,
only when loads are curtailed as shown Fig. 13.

259
2004 IEEE International Conference on Electric Utility Deregulation, Restructuring and Power Technologies (DFVT2004) April 2004 Hong Kong

0 Oll-fired Cl LNG-fired Coalfired

Nuclear El Hydro

:2"
0
n
1 6160
Hour L h 1

8760
Duration Hours [hl Duration HOUN[hl Fig. 15 Load and quantity of transaction duration curve
(a) VOLL pncing (b) Reserve operating
Fig. 12 Load Duration Curve Consequently, it could be proven that the profits of peak
plants would be assured even if the price cap would be set up
relatively low, and that market risk could be decreased if
300 1 volatility of electricity price could be reduced in operating
1
2 250
f 200 R VOLL Pricing
reserve. Moreover, it can be expected that operating reserve and
VOLL pricing can increase supply reliability because there are
incentives to install peak power plants more as supply entities
Operating Reserve
can profit at low-risk. However, customers have to pay more
electricity costs to hedge market risk. As stated above, it could
be pointed out that we should take into consideration also effects
0 of risk-hedge to assure supply reliability in the operating reserve
1 8760 and VOLL pricing.
Duration Hours [h]
Fig. 13 Price Duration Curve in Real-time market D. Efsects of Unit Commitment Costs on Bidding Price
Next, we added UC (unit commitment) costs to the action-
OSslcs .Vanable Cost DFixsd Cost .Profit1
bsales DVanabie rmF'xed
value functions of three types of thermal power plant agents.
The UC costs are listed in Table 11. In this case, we assume that
y equals to 0.5 because we have to distinguish two different
states of generators, i.e. operating and idling. The power
generation mixes for a certain week in winter for the two cases,
one case is the case where UC cost is not added and the other
one is the case where UC cost is added, are shown in Fig. 16, and
the average bid price ugl by plant type is shown in Fig. 17.
+sa *," 3." ob ++so ,$& $ $6 ob

TABLE I1
(a) VOLL pricing (b) Operating reserve
COSTS OF PLANTS TO START OR STOP
Fig.14 Profits per Plants Plant I Coal LNG Oil
Stan-cost I
The comparison of the profits per kW of each type of power
StoPcost I "
plants between the two cases is shown in Fig.14. In the VOLL
pricing, the sales income of peak plant (e.g. oil-fired power
plant) is lower than others, so it can just manage to collect its
fixed cost. The profit of this power plant in an actual market can
be probably lower than that of this model, because electric
supply and demand balance does not necessarily become so
stringent annually. The price cap can be raised to higher value,
in order to assure electric supply for peak load. Due to high
volatility in electricity prices of the real time market, however, it
is unlikely that a sufficient operating reserve rate can be assured.
On the other hand, oil-fired power plant increases significantly
its profit in the operating reserve. This is because oil-fired power (a) No start-cost, stop-cost (b) Add in start-cost, stop-cost
plant for peak load increases quantity of transaction for assuring Fig. 16. The power generation mixes
operating reserve rate.
Load and quantity of transaction duration curves for one year We cannot recognize noticeable differences between the two
of each plant in the operating reserve are shown in Fig 15. It is graphs in Fig.16. In Fig.17, however, we can find significant
clear that peak power plant as oil-fired power plant make a differences in the bid prices of oil-fired and LNG-fired power
successful bid and it can increase quantity of transaction for plants constructed for peak and middle loads, respectively.
assuring a supply capability in the operating reserve as shown in LNG-fired power plant agents keep their bid prices lower,
Fig.15. presumably because there is strong negative correlation between
their bid prices and amounts of electricity they can sell. The
smaller electricity sales lead to the lower capacity usage rates.
The lower capacity usage rates are supposed to increase the

260
2004 IEEE International Conference on Electric Utility Deregulation, Restructuring and Power Technologies (DFWT2004) April 2004 Hong Kong

occasion of unit commitment of the power plants with additional [4] Janusz W. Bialek, “Gaming the Uniform-Price Spot Market:
costs. On the other hand, oil-fired power plants force up their bid Quantitive Analysis”, IEEE Trans. on Power Systems, Vol. 17, No.
prices. These peak load power plants cannot avoid the state 3, Aug 2002
[SI Shimomura, S. et al., “Analysis of the Pricing Process in
transition between running and idling in all the cases, as shown Electricity Market using Multi-Agent Model, IEEJ.EIS, Feb.2004
in Fig.17. Because there seems to be almost no correlation to be appeared in Japanese
between the bid prices and amounts of electricity sales for oil-
fired power plant, the higher bid prices of oil-fired power plants VII. BIOGRAPHIES
therefore do not necessarily reduce their economic benefit
significantly. The introduction of the UC cost might have
uneven impacts on the bidding strategies among power plant Yasumasa Fujii graduated from University of
Tokyo in 1988. He received master and doctor
agents of different type. dcgrees of Engineering from University of
Tokyo in 1990 and 1993, respectively. He is
113 no cost add in cos4 113 no cost R add in cos4 Associate Professor at Department of Electrical
ino , I 25 t I Engineering, Graduate School of Engincering,
University of Tokyo. He is a member of IEEE
and the Institute of Electrical Engineers of Japan.
His research interests arc modeling and analysis
of energy systems.

Tomoaki Okamura graduated from Tokyo

coal LNG oil coal LNG oil Metropolitan University in 2003. He entered
master course of Department of Electrical
(a) At off peak time (b) At peak time Engineering, Graduate School of Engineering,
Fig. 17. Bid price of each plant at peak time or off peak time University of Tokyo. He is a student mcmber of
the Institute of Electrical Engineers of Japan.
His research interest is the analysis of electricity
market and transmission congestions with multi-
V. CONCLUSIONS agent simulation model.
The model based on multi-agent with learning enables us to
analyze more general electricity markets, which have several Kenji Inagaki graduated from Waseda
different types of power plants with seasonal and hourly demand University in 2003. He entered master course of
fluctuation, real-time market and operating reserve market. In Department of Advanced Energy, Graduate
this paper, specifically we analyzed the market powers of some School of Frontier Science, University of Tokyo.
type of power generation plants, i.e. hydro power, nuclear power, He is a student member of the Institute of
coal-fired power, LNG-fired power and oil-fired power, by Electrical Engineers of Japan. His research
intercst is the analysis of electricity market and
changing the number of agents of each type of plants from one optimal generation mix with multi-agent
to ten. We assumed only one agent of the power plant in a so- simulation model.
called monopolistic case, and ten agents in a competitive case.
In the paper, several numerical simulation results, such as price
Kenji Yamaji graduated from University of
duration curves of the modeled market, were discussed. The Tokyo in 1972. He received master and doctor
simulation result suggested that the small number of peak load dcgrces of Engineering from University of
power plants, such as oil-fired power plants, may lead to Tokyo in 1974 and 1977, respectively. He is
significant price increases in real time market, and it highlights Professor at Department of Advanced Energy,
the necessity to care about their market shares. Through the use Graduate School of Frontier Science, University
of Tokyo. He is a member of the Institute of
of our multi-agent model of electricity market, we also evaluated Electrical Engineers of Japan. His research
the roles of operating reserve markets and price caps in real time interests are assessment of climate change
markets. The simulation results of our model indicated that the technologies and policies, cncrgy tcchnology
economic viability of peak load power plants, which seldom assessment and advanced management of electricity network.
have a chance to operate in the so-called power exchange market,
is improved significantly by the introduction of an operating
reserve market. The results also indicated that the introduction
of the UC cost might have uneven impacts on the bidding
strategies among power plant agents of different type.

VI. REFERENCES
James D. Wcbcr and Thomas J. Overbye : “An Individual Wclfare
Maximization Algorithm for Electricity Markets”, IEEE Trans. on
Power Systems, Vol. 17, No. 3, Aug. 2002
G. R. Gajjar, S. A. Khaparde, P. Nagaraju, and S. A. Soman :
“Application of Actor-Critic Lcaming Algorithm for Optimal
Bidding Problem of a Genco”, IEEE Trans. on PoM’erSvsteins, Vol.
18, No. I , Feb. 2003
Benjamin F. Hobbs, Carolyn B. Metzler, and Jong-Shi Pang,
“Strategic Gaming Analysis for Electric Powcr Systems: An MPEC
Approach” IEEE Trans. on Power Systems, Vol. 15, No. 2, May
2000

26 1