Internet Traffic Engineering

by Optimizing OSPF Weights

Bernard Fortz Mikkel Thorup
Service de MathCmatiques de la Gestion AT&T Labs-Research
Institut de Statistique et de Recherche Op6rationnelle Shannon Laboratory
UniversitC Libre de Bruxelles Florham Park, NJ 07932
Brussels, Belgium USA
bfortz@smg.ulb.ac.be mthorup@research.att.com

Abstract-Open Shortest Path First (OSPF) is the most commonly used nodes and arcs represent routers and the links between them.
intra-domain internet routing protocol. ’Rafficflow is routed along shortest Each arc a has a capacity .(U) which is a measure for the amount
paths, splitting flow at nodes where several outgoing links are on shortest
paths to the destination. The weights of the links, and thereby the shortest of traffic flow it can take. In addition to the capacitated network,
patb routes, can be changed by the network operator. The weights could we are given a demand matrix D that for each pair (6, t ) of nodes
be set proportional to their physical distances, but often the main goal is to tells us how much traffic flow we will need to send from s to t.
avoid congestion, i.e. overloading of links,and the standard heuristic rec- We shall refer to s and t as the source and the destination of the
ommended by Cwo is to make the weight of a link inversely proportional
to its capacity. demand. Many of the entries of D may be zero, and in particu-
Our startiug point was a proposed AT&T WorIdNet backbone with de- lar, D ( s ,t ) should be zero if there is no path from s to t in G.
mands projected from previous measurements. The desire was to optimize The routing problem is now, for each non-zero demand D(s,t ) ,
the weight setting based on the projected demands. We showed that opti-
m i z i i the weight settings for a given set of demands is NP-hard, so we re- to distribute the demanded flow over paths from s to t. Here, in
sorted to a local search heuristic. Surprisingly it turned out that for the pro- the general routing problem, we assume there are no limitations
posed AT&T WorlWet backbone, we found weight settingsthat performed to how we can distribute the flow between the paths from s to t .
within a few percent from that of the optimal general routing where the Bow The above definition of the general routing problem is equiv-
for each demand is optimally distributed over all paths between source and
destination. This contrasts the common belief that OSPF routing leads to alent to the one used e.g. in Awduche et al. [ 11. Its most con-
congestion and it shows that for the network and demand matrix studied troversial feature is the assumption that we have an estimate of
we cannot get a substantially better load balancing by switching to the pro- a demand matrix. This demand matrix could, as in our case for
posed more flexible Multi-protocol Label Switching (MPLS) technologies.
Our techniques were also tested on synthetic internetworks, based on a the proposed AT&T WorldNet backbone, be based on concrete
model of Zegura et al. (INFOCOM’96), for which we did not always get measures of the flow between source-destinationpairs. The de-
quite as close to the optimal general routing. However, we compared with mand matrix could also be based on a concrete set of customer
standard heuristics, such as weights inversely proportional to the capac-
ity or proportional to the physical distances, and found that, for the same
subscriptions to virtual leased line services. Our demand matrix
network and capacities, we could support a 50%-110% increase in the de- assumption does not accommodate unpredicted bursts in traffic.
mands. However, we can deal with more predictable periodic changes,
Our assumed demand matrix can also be seen as modeling service level say between morning and evening, simply by thinking of it as
agreements (SLAs) with customers, with demands representing guarantees
of throughput for virtual leased lines. two independent routing problems: one for the morning, and
Keywords-OSPF, MPLS, traffic engineering, local search, hashing ta- one for the evening.
bles, dynamic shortest paths, multi-commodity network Rows. Having decided on a routing, the load [ ( U ) on an arc a is the
total flow over a, that is [ ( a ) is the sum over all demands of
I. INTRODUCTION the amount of flow for that demand which is sent over a. The
utilization of a linka is e ( a ) / c ( a ) .
P ROVISIONING an Internet Service Provider (ISP) back-
bone network for intra-domain IP traffic is a big challenge,
particularly due to rapid growth of the network and user de-
Loosely speaking, our objective is to keep the loads within the
capacities. More precisely, our cost function @ sums the cost of
the arcs, and the cost of an arc U has to do with the relation
mands. At times, the network topology and capacity may seem . our experimental study, we had
between t ( a ) and ~ ( u )In
insufficient to meet the current demands. At the same time,
there is mounting pressure for ISPs to provide Quality of Ser-
vice (QoS) in terms of Service Level Agreements (SLAs) with aEA
customers, with loose guarantees on delay, loss, and throughput.
All of these issues point to the importance of tr@c engineering, where for all a E A , Ga(0) = 0 and
making more efficient use of existing network resources by tai- ( 1 for 0 5 X / C ( U ) < 1/3
loring routes to the prevailing traffic. 3 for 1/3 5 Z / C ( U ) < 2/3
10 for 2/3 5 z / c ( u ) < 9/10
A. The general routing problem dh(4 = 70 for 9/10 5 z / c ( u ) < 1
The getzeral routing problem is defined as follows. Our net- 500 for 15 ./.(a) < 11/10
work is a directed graph, or multi-graph, G = ( N ,A ) whose 5000 for 11/10 5 z / c ( u ) < 00

(c) 2000 IEEE

0-7803-5880-5/00/$10.00 519 IEEE INFOCOM 2000
 The quality of OSPF routing depends highly on the choice of
weights. Nevertheless, as recommended by Cisco [3], these are
often just set inversely proportional to the capacities of the links,
without taking any knowledge of the demand into account.
It is widely believed that the OSPF protocol is not flexible
enough to give good load balancing as defined, for example, in
our objective function. This is one of the reasons for introducing
the more flexible Multi-protocol Label Switching (MPLS) tech-
nologies ([l]. [4]). With MPLS one can in principle decide the
path for each individual packet. Hence, we can simulate a so-
fig. 1. Arccost @,(!(a)) asafunctionofloadt(a) forarccapacityc(a) = 1.
lution to the general routing problem by distributing the packets
on the paths between a source-destination pair using the same
The function !Da is illustrated in Fig. 2. The idea behind Qa is distribution as we used for the flow. The MPLS technology has
that it is cheap to send flow over an arc with a small utilization. some disadvantages. First of all, MPLS is not yet widely de-
As the utilization approaches loo%, it becomes more expensive, ployed, let alone tested. Second OSPF routing is simpler in the
for example because we get more sensitive to bursts. If the uti- sense that the routing is completely determined by one weight
lization goes above loo%,we get heavily penalized, and when for each arc. That is, we do not need to make individual routing
the utilization goes above 110%the penalty gets so high that this decisions for each sourcddestination pair. Also, if a link faild,
should never happen. the weights on the remaining links immediately determines the
The exact definition of the objective function is not so im- new routing.
portant for our techniques, as long as it is a a piece-wise linear
increasing and convex function. C. Our results
PropositionI: If each @a is a piece-wise linear increasing The general question studied in this paper is: Can a suj-
and convex function, we can solve the general routing problem jiciently clever weight settings make OSPF muting perfo+
optimally in polynomial time. nearly as well as optimal generaUMPL3 routing? i

Knowing the optimal solution for the general routing problem Our first answer is negative: for arbitrary n, we construct f
is an important benchmark for judging the quality of solutions instance of the routing problem on 8 n3 nodes where any OSPF
based on, say, OSPF routing. routing has its average flow on arcs with utilization Q ( n )timds
The above objective function provides a general best effort higher than the max-utilization in an optimal general solution. 1
measure. We have previously claimed that our approach can also With our concrete objective function, this demonstrates a gap
be used if the demand matrix is modeling service level agree- of a factor approaching 5000 between the cost of the optimal
ments (SLAs) with customers, with demands representing guar- general routing and the cost of the optimal OSPF routing. I
antees of throughput for virtual leased lines. In this case, we The next natural question is: how well does OSPF routing I
are more interested in ensuring that no packet gets sent across perform on real networks. In particular we wanted to answer
overloaded arcs, so our objective is to minimize a maximum this question for a proposed AT&T WorldNet backbone. In addi-
rather than a sum over the arcs. However, due to the very high tion, we studied synthetic intemetworks, generated as suggested
penalty for overloaded arcs, our objective function favours solu- by Calvert, Bhattacharjee, Daor, and Zegura 151, [6].Finding a
tions without overloaded arcs. A general advantage to working perfect answer is hard in the sense that it is NP-hard to find an
with a sum rather than a maximum is that even if there is a bot- optimal setting of the OSPF weights for an arbitrary network.
tleneck link that is forced to be heavily loaded, our objective Instead we resorted to a local search heuristic, not guaranteed
function still cares about minimizing the loads in the rest of the to find optimal solutions. Very surprisingly, it turned out that
network. for the proposed AThT WorldNet backbone, the heuristic found
weight settings making OSPF routing performing within a few
B. OSPF versus MPLS routing protocols percent from the optimal general routing. Thus for the proposed
Unfortunately, most intra-domain internet routing protocols AT&T WorldNet backbone with our projected demands, y d
today do not support a free distribution of flow between source with our concrete objective function, there would be no sub-
stantial traffic engineering gain in switching from the existing I
and destination as defined above in the general routing problem.
The most common protocol today is Open Shortest Path First well-tested and understood robust OSPF technology to the nefw
I
(OSPF) [2]. In this protocol, the network operator assigns a MPLS alternative.
weight to each link, and shortest paths from each router to each For the synthetic networks, we did not always get quite t
as
destination are computed using these weights as lengths of the close to the optimal general routing. However, we compared o b
links. In each router, the next link on all shortest paths to all local search heuristic with standard heuristics, such as weights
possible destinations is stored in a table, and a demand going inversely proportional to the capacities or proportional to the
in the router is sent to its destination by splitting the flow be- physical distances, and found that, for the same network abd
tween the links that are on the shortest paths to the destination. capacities, we could support a 50%-110% increase in the de-
The exact mechanics of the splitting can be somewhat compli- mands, both with respect to our concrete cost function and, si-
cated, depending on the implementation. Here, as a simplifying multaneously, and with respect to keeping the max-utilization
approximation, we assume that it is an even split. below 100%.

0-7803-5880-5/00/$10.00 (c) 2000 IEEE 520 IEEE INFOCOM 2000

D. Technical contributions lated as the following linear program.

Our local search heuristic is original in its use of hash tables mina =E
,EA
to avoid cycling and for search diversification. A first attempt of
using hashing tables to avoid cycling in local search was made subject to
by Woodruff and Zemel [7], in conjunction with tabu search.
Our approach goes further and avoids completely the problem
specific definitions of solution attributes and tabu mechanisms,
leading to an algorithm that is conceptually simpler, easier to
implement, and to adapt to other problems.
Our local search heuristic is also original in its use of more ad- a E A,
vanced dynamic graph algorithms. Computing the OSPF rout- (8,t)ENxN
ing resulting from a given setting of the weights tumed out to @a L e(a) a E A, (3)
be the computational bottleneck of our local search algorithm,
as many different solutions are evaluated during a neighborhood 0,2 se(4 - $ c ( u ) a E A, (4)
exploration. However, our neighborhood structure allows only oa 2 ioe(a) - g+) a EA, (5)
a few local changes in the weights. We therefore developed ef- 0,2 7 0 t ( a )- ?.(a) a E A, (6)
ficient algorithms to update the routing and recompute the cost
of a solution when a few weights are changed. These speed-ups 0, 2 500l(a)- y c ( a ) a E A, (7)
are critical for the local search to reach a good solution within @a 2 50ooe(~)- Y C ( Ua)E A , (8)
reasonable time bounds. >0
fa
(8J)
- a € A; s , t E N . (9)
Constraints (I) are flow conservation constraints that ensure the
E. Contents desired traffic flow is routed from s to t , constraints (2) define
the load on each arc and constraints (3) to (8) define the cost on
In Section I1 we formalize our general routing model as a lin- each arc.
ear program, thereby proving Proposition 1. We present in Sec- The above program is a complete linear programming for-
tion I11 a scaled cost function that will allow us to compare costs mulation of the general routing problem, and hence it can be
across different sizes and topologies of networks. In Section IV. solved optimally in polynomial time (JShachiyan [SI), thus set-
a family of networks is constructed, demonstrating a large gap tling Proposition 1. In our experiments, we solved the above
between OSPF and multi-commodity flow routing. In Section V problems by calling CPLEX via AMPL. We shall use @OPT to
we present our local search algorithm, guiding the search with denote the optimal general routing cost.
hash tables. In Section VI we show how to speed-up the calcula-
tions using dynamic graph algorithms. In Section VII, we report B. OSPF routing
the numerical experiments. Finally, in Section VI11 we discuss
In OSPF routing, we choose a weight w(a) for each arc.
our findings.
The length of a path is then the sum of its arc weights, and
Because of space limitations, we defer to the journal version we have the extra condition that all flow leaving a node aimed
the proof that it is NP-hard to find an optimal weight setting at a given a destination is evenly spread over arcs on short-
for OSPF routing. In the journal version, based on collaborative est paths to that destination. More precisely, for each source-
work with Johnson and Papadimitriou, we will even prove it NP- destination pair (s,t) E N x N and for each node I, we have
hard to find a weight setting getting within a factor 1.77 from that f[:;) = 0 if (t, y) is not on a shortest path from s to t ,
optimality for arbitrary graphs.
and that f[i:i\ = f[;:%, if both (I,y) and (2,y') are on short-
est paths from s to t. Note that the routing of the demands is
11. MODEL completely determined by the shortest paths which in turn are
determined by the weights we assign to the arcs. Unfortunately,
A. Optimal routing the above condition of splitting between shortest paths based on
variable weights cannot be formulated as a linear program, and
Recall that we are given a directed network G = (NIA ) with this extra condition makes the problem NP-hard.
a capacity C ( U ) for each a E A . Furthermore, we have a demand We shall use O o p t ~ stopdenote
~ the optimal cost with OSPF
matrix D that for each pair (s,t ) E N x N tells the demand routing.
D ( s ,t ) in traffic flow between s and t. We will sometimes re-
fer to the non-zero entries of D as the demands. With each pair 111. NORMALIZING
COST
(s,t ) and each arc a, we associate a variable telling how
fiSlt)
We now introduce a normalizing scaling factor for the cost
much of the traffic flow from s to t goes over a. Variable )(a) function that allows us to compare costs across different sizes
represents the total load on arc a, i.e. the sum of the flows go- and topologies of networks. To define the measure, we introduce
ing over a , and @, is used to model the piece-wise linear cost
function of arc a. @Uncop = C ( D ( s , t ).distl(s,t)). (10)
With this notation, the general routing problem can be formu- ( s , t ) E Nx N

(c) 2000 IEEE

0-7803-5880-5/00/$10.00 521 IEEE INFOCOM 2000
Above distl(.) is distance measured with unit weights (hop
count). Also, we let @ u , i t ~ denote ~ p ~ the cost of OSPF rout-
ing with unit weights. Below we present several nice properties
of @uncap and CPUnjtOSPF. It is (ii) that has inspired the name
"Uncapacitated".
Lemma 2:
(i) @uncap is the total load if all traffic flow goes along unit
weight shortest paths.
(ii) @uncap= cPun;tOSpF if all arcs have unlimited capacity.
i
OP 01
",,,oz.,,,- - ' 1 -
Fig. 2. Arc packet cost as function of utilization
(iii) @uncapis the minimal total load of the network.
(iv) @uncapL @OPT.
(VI Q u n i t O S P F < 5000 . @uncap.
Above, 5000 is the maximal value of CP;.
PmoJ Above (i) follows directly from the definition. Now
(ii) follows from (i) since the ratio of cost over load on an arc
is 1 if the capacity is more than 3 times the load. Further (iii)
follows from (i) because sending flow along longer paths only
increases the load. From (iii) we get (iv) since 1 is the smallest
possible ratio of cost over load. Finally we get (v) from (ii) since
decreasing the capacity of an arc with a given load increases the
arc cost with strictly less than a factor 5000 if the capacity stays I
positive. I
Hg. 3. Gap between general flow solution on the left and OSPF solution on thei
Our scaled cost is now defined as: right !
i
others going below, or by the packet following a longer detouri
From Lemma 2 (iv) and (v). we immediately get: using less congested arcs. Nevertheless, it is natural to say thad
a routing congests a network if cP* 2 10;. 1
i
Iv. MAXIMAL
GAP BETWEEN OPT AND OSPF
Note that if we get @* = 1, it means that we are routing along
unit weight shortest paths with all loads staying below 1/3 of the From (12), it follows that the maximal gap between optimal
capacity. In this ideal situation there is no point in increasing the general routing and optimal OSPF routing is less than a factor
capacities of the network. 5000. In Lemma 3, we will now show that the gap can in fact apd
proach 5000. Our proof is based on a construction where OSPF
A packet level view leads to very bad congestion for any natural definition of conges-
Perhaps the most instructive way of understanding our cost tion. More precisly, for arbitrary n, we construct an instances of
functions is on the packet level. We can interpret the demand the routing problem on M n3 nodes with only one demand, and
D ( s ,t) as measuring how many packets -of identical sizes - where all paths from the source to the destination are shortest
we expect to send from s to t within a certain time frame. Each with respect to unit weights, or hop count. In this instance, any
packet will follow a single path from s to t , and it pays a cost OSPF routing has its average flow on arcs with utilization Q ( n )
for each arc it uses. With our original cost function '9, the per times higher than the max-utilization in an optimal general solu-
packet cost for using an arc is 0, (e(.))/e( U ) . Now, CP can be tion. With our concrete objective function, this demonstrates a
obtained by summing the path cost for each packet. gap of a factor approaching 5000 between the cost of the optimal
The per packet arc cost @a(!(Cl))/e(Q) is a function q5 of general routing and the cost of the optimal OSPF routing.
Lemma 3: There is a family of networks G, so that the opti,
the utilization L ( a ) / c ( u ) .The function 4 is depicted in Fig. 2.
First is 1 while the utilization is 5 1/3. Then q5 increases to
mal general routing approaches being 5000 times better than thF r
10 - 16/3 = l o $ for a full arc, and after that it grows rapidly optimal OSPF routing as n + 00. ?
1
towards 5000. The costfactor of a packet is the ratio of its cur- Proof: In our construction, we have only one demand wi$
rent cost over the cost it would have paid if it could follow the source s and destination t , that is, D ( z ,y) > 0 if and only !f
unit-weight shortest path without hitting any utilization above (cIy) = ( s , t ) . The demand is n. Starting from s, we have :a
1/3, hence paying the minimal cost of 1 for each arc traversed. directed path P = (sI 2 1 I ... 2,) where each arc has capacity
The latter ideal situation is what is measured by @uncap, and 372. For each i, we have a path Qj of length n2 - i from z i to t .
therefore CP* measures the weighted average packet cost factor Thus, all paths from s to t have length n2.Each arc in each Q;
where each packet is weighted proportionally to the unit-weight has capacity 3. The graph is illustrated in Fig. 3 with P being
distance between its end points. the thick high capacity path going down on the left side.
If a packet follows a shortest path, and if all arcs are exactly In the general routing model, we send the flow down P , let-
full, the cost factor is exactly lo$. The same cost factor can ting 1 unit branch out each Q i . This means that no arc gets
of course be obtained by some arcs going above capacity and more flow than a third of its capacity, so the optimal cost @zpT
(c) 2000 IEEE
0-7803-5880-5/00/$10.00 522 IEEE lNFOCOM 2000
satisfies Single weight change. This simple modification consists in
@zPT 5 n(n2+ n ) = (1+ o(1))n3. changing a single weight in w. We define a neighbor w' of w
for each arc Q E A and for each possible weight t E W\{wa)
In the OSPF model, we can freely decide which Qiwe will by setting wb = t and wl, = W b for all b # a.
use, but because of the even splitting, the first Qiused will get Evenly balancingjlows. Assuming that the cost function Qa ()
half the flow, i.e. n / 2 units, the second will get n/4 units, etc. for an arc Q E A is increasing and convex, meaning that we
Asymptoticallythis means that almost all the flow will go along want to avoid highly congested arcs. we want to split the flow as
arcs with load a factor Q ( n ) above their capacity, and since evenly as possible between different arcs.
all paths uses at least n2 - n arcs in some Qthe i, OSPF cost More precisely, consider a demand node t such that
(P;;l,tOSPF satisfies CJEN D(s,t) > 0 and some part of the demand going to t
goes through a given node 2. Intuitively, we would like OSPF
@zptOSPF
2 ( 1 - o(1))5000n3. routing to split the flow to t going through t evenly along arcs
leaving t. This is the case if all the arcs leaving 2 belong to a
We conclude that the ratio of the OSPF cost over the optimal shortest path from z to t. More precisely, if 21 , z2,. . . , x p are
cost is such that
00.
'%izF 2 ( 1 - o( 1))5000 --+ 5000 as n 3
m
the nodes adjacent to 2, and if Pi is one of the shortest paths
from xi to t , for i = 1, . . . ,p , as illustrated in Fig. 4,then we
want to set w' such that
V. OSPF WEIGHT SETTING USING LOCALSEARCH
A. Overview of the local search heuristic
W{Z,Zl) + w'(P1) = W{Z,Z2) + w">
-
- ...--.
+4 P P ) .
In OSPF routing, for each arc Q E A, we have to choose
a weight wa. These weights uniquely determine the shortest where ~ ' ( p denotes
i) the sum of the weights of the arcs belong-
paths, the routing of traffic flow, the loads on the arcs, and fi- ing to Pi. A simple way of achieving this goal is to set
nally, the cost function a. In this section, we present a lo-
w* - w(Pj) if a = (x,ti),fori = 1,.. . , p ,
cal search heuristic to determine a weights vector ( W a ) a E A that
minimizes 0. We let W := { 1, . . . ,wmaz} denote the set of
w; =
{ otherwise.
possible weights.
Suppose that we want to minimize a function f over a set
+
where w* = 1 maxi=1, , p { ~ ( P , ) } .
A drawback of this approach is that an arc that does not belong
X of feasible solutions. Local search techniques are iterative to one of the shortest paths from t to t may already be con-
procedures in which a neighborhood N ( x ) X is defined at gested, and the modifications of weights we propose will send
each iteration for the current solution x E X, and which chooses more flow on this congested arc, an obviously undesirable fea-
the next iterate x' from this neighborhood. Often we want the ture. We therefore decided to choose at random a threshold ratio
neighbor x' E n/(x) to improve on f in the sense that f(z') < 0 between 0.25 and 1, and we only modify weights for arcs in
f (2). the maximal subset B of arcs leaving x such that
Differences between local search heuristics arise essentially
from the definition of the neighborhood, the way it is explored, w(c,zc) + pi) I ~ ( z , z + ~ )w(pj) V i E B , j B,
and the choice of the next solution from the neighborhood. De- l ( s , z , ) ( ~ <_ , V i E B.
) 0 ~ ( zzi)
scent methods consider the entire neighborhood, select an im-
proving neighbor and stop when a local minimum is found. In this way, flow going from n to t can only change for arcs
Meta-heuristics such as tabu search or simulated annealing al- in B , and choosing 0 at random allows to diversify the search.
low non-improving moves while applying restrictions to the Another drawback of this approach is that it does not ensure
neighborhood to avoid cycling. An extensive survey of local that weights remain below wmaz. This can be done by adding
search and its application can be found in Aarts and Lenstra [9]. the condition that ma?riEBw(Pi) - miniEB W ( P i ) 5 Wmaz is
In the remainder of this section, we first describe the neigh- satisfied when choosing B.
borhood structure we apply to solve the weight setting problem.
Second, using hashing tables, we address the problem of avoid- C. Guiding the search with hashing tables
ing cycling. These hashing tables are also used to avoid repeti- The simplest local search heuristic is the descent method that,
tions in the neighborhood exploration. While the neighborhood at each iteration, selects the best element in the neighborhood
search aims at intensifying the search in a promising region, it is and stops when this element does not improve the objective
often of great practical importance to search a new region when function. This approach leads to a local minimum that is of-
the neighborhood search fails to improve the best solution for a ten far from the optimal solution of the problem, and heuris-
while. These techniques are called search diversification and are tics allowing non-improving moves have been considered. Un-
addressed at the end of the section. fortunately, non-improving moves can lead to cycling, and one
must provide mechanisms to avoid it. Tabu search algorithms
B. Neighborhood structure (Glover [IO]), for example, make use of a tabu list that records
A solution of the weight setting problem is completely char- some attributes of solutions encountered during the recent itera-
acterized by its vector w of weights. We define a neighbor tions and forbids any solution having the same attributes.
wr E N ( w ) of w by one of the two following operations ap- We developed a search strategy that completely avoids cy-
plied to w. cling without the need to store complex solution attributes. The

0-7803-5880-j/00/$10.00 ( c )2000 IEEE 523 IEEE INFOCOM 2000

 the current solution is improved, we divide the size of the sam-
pling by 3, while we multiply it by 10 each time the current so-
lution is not improved. Moreover, we enforce sampling at least
1 % of the neighborhood.

E. Diversocation
Another important ingredient for local search efficiency is di-
versification. The aim of diversification is to escape from re-
gions that have been explored for a while without any improve-
ment, and to search regions as yet unexplored.
In our particular case, many weight settings can lead to the
same routing. Therefore, we observed that when a local min-
imum is reached, it has many neighbors having the same cost,
Fig. 4. The second type of move tries to make all paths form a to t of equal
length.
leading to long series of iterations with the same cost value. To
escape from these "long valleys" of the search space, the sec-
ondary hashing table is again used. This table is generally reset
solutions to our problem are 1.1 [-dimensional integer vectors. at the end of each iteration, since we want to avoid repetitions
Our approach maps these vectors to integers, by means of a inside a single iteration only. However, if the neighborhood ex-
hashing function h, chosen as described in [ll]. Let I be the ploration does not lead to a solution better than the current one,
number of bits used to represent these integers. We use a we do not reset the table. If this happens for several iterations,
boolean table T to record if a value produced by the hashing more and more collisions will occur and more potentially good
function has been encountered. As we need an entry in T for solutions will be excluded, forcing the algorithm to escape from
each possible value returned by h, the size of T is 2'. In our the region currently explored. For these collisions to appear at a
implementation,I = 16. At the beginning of the algorithm, all reasonable rate, the size of the secondary hashing table must be
entries in T are set to false. If w is the solution produced at a small compared to the primary one. In our experiments, its size
given iteration, we set T(h(w))to true, and, while searching the is 20 times the number of arcs in the network.
neighborhood, we reject any solution w' such that T ( h ( w ' ) )is This approach for diversification is useful to avoid regions
true. with a lot of local minima with the same cost, but is not suffi-
This approach completely eliminates cycling, but may also cient to completely escape from one region and go to a possibly
reject an excellent solution having the same hashing value as more attractive one. Therefore, each time the best solution found
a solution met before. However, if h is chosen carefully, the is not improved for 300 iterations, we randomly perturb the cur- '
probability of collision becomes negligible. A first attempt of rent solution in order to explore a new region from the search :
using hashing tables to avoid cycling in local search was made space. The perturbation consists of adding a randomly selected i
by Woodruff and Zemel [ 7 ] , in conjunction with tabu search. perturbation, uniformly chosen between -2 and +2, to 10 % of
It differs from our approach since we completely eliminate the the weights.
tabu lists and the definition of solution attributes, and we store
the values for all the solutions encountered, while Woodruff and VI. COST EVALUATION
Zemel only record recent iterations (as in tabu search again). We will now first show how to evaluate our cost function
Moreover, they store the hash values encountered in a list, lead- for the static case of a network with a specified weight set-
ing to a time linear in the number of stored values to check if a ting. Computing this cost function from scratch is unfortunately
solution must be rejected, while with our boolean table, this is too time consuming for our local search, so afterwards, we will
done in constant time. show how to reuse computations, exploiting that there are only
few weight changes between a current solution and any solution
D. Speeding up neighborhood evaluation in its neighborhood.
Due to our complex neighborhood structure, it turned out that
A. The static case
several moves often lead to the same weight settings. For ef-
ficiency, we would like to avoid evaluation of these equivalent We are given a directed multigraph G = ( N , A ) with arc
moves. Again, hashing tables are a useful tool to achieve this capacities { C a } a c A , demand matrix D,and weights { W a } a c A .
goal : inside a neighborhood exploration, we define a secondary For the instances considered, the graph is sparse with IAl =
hashing table used to store the encountered weight settings as 0 (IN I). Moreover, in the weighted graph the maximal distance,
above, and we do not evaluate moves leading to a hashing value between any two nodes is O( INI).
already met. We want to compute our cost function a. The basic problem
The neighborhoodstructure we use has also the drawback that is to compute the loads resulting from the weight setting. Wel
the number of neighbors of a given solution is very large, and will consider one destination t at a time, and compute the totali
exploring the neighborhood completely may be too time con- flow from all sources s E N to t . This gives rise to a certain/
fLs")
suming. To avoid this drawback, we only evaluate a randomly partial load 1: = C s E N for each arc. Having done th4
selected set of neighbors. above computation for each destination t , we can compute the
We start by evaluating 20 % of the neighborhood. Each time load 1, on arc a as 1;. ztGN
(c) 2000 IEEE
0-7803-5880-5/00/$10.00 524 IEEE INFOCOM 2000
 To compute the flow to t , our first step is to use Dijkstra’s To see that the above suffices, first note that the nodes visited
algorithm to compute all distances to t (normally Dijkstra’s al- are considered in order of decreasing distances. This follows
gorithm computes the distances away from some source, but we because we always take the node at the maximal distance and
can just apply such an implementation of Dijkstra’s algorithm because when we add a new node z to M , it is closer to t than
to the graph obtained by reversing the orientation of all arcs in the currently visited node y. Consequently, our dynamic algo-
G). Having computed the distance cl: to t for each node, we rithm behaves exactly as our static algorithm except that it does
compute the set At of arcs on shortest paths tot, that is, not treat nodes not in M . However, all nodes whose incoming or
outgoing arc set changes, or whose incoming arc loads change
are put in M , so if a node is skipped, we know that the loads
around it would be exactly the same as in the previous evalua-
For each node 2, let IS: denote its outdegree in A t , i.e. d: = tion.
I{Y E N (X,Y) E A t N .
Observation 4: For all (y, z ) E A t , VII. NUMERICAL EXPERIMENTS
We present here our results obtained with a proposed AT&T
WorldNet backbone as well as synthetic internetworks.
Besides comparing our local search heuristic (HeurOSPF)
Using Observation 4,we can now compute all the loads l{y,z, with the general optimum (OPT), we compared it with OSPF
as follows. The nodes y E N are visited in order of decreas- routing with “oblivious” weight settings based on properties of
ing distance 4 to 1. When visiting a node y. we first set the arc alone but ignoring the rest of the network. The oblivi-
= k(D(Y>t) $- C ( z , y ) E A * ‘ t ~ , y ) ) ’ Second we set 2tg,,) = ous heuristics are InvCapOSPF setting the weight of an arc in-
versely proportional to its capacity as recommended by Cisco
for each (y, z ) E At.
[3], UnitOSPF just setting all arc weights to 1, L2OSPF setting
To see that the above algorithm works correctly, we note the
the weight proportional to its physical Euclidean distance ( L z
invariant that when we start visiting a node y, we have correct
norm), and RandomOSPF, just choosing the weights randomly.
loads 1; on all arcs a leaving nodes coming before y. In partic-
Our local search heuristic starts with randomly generated
ular this implies that all the arcs (2 y) entering y have correctly
weights and performs 5000 iterations, which for the largest
computed loads Ifx,Y), and hence when visiting y, we compute
graphs took approximately one hour. The random starting point
the correct load ltY,=) for arcs (y, z) leaving y. was weights chosen for RandomOSPF, so the initial cost of our
Using bucketingfor the priority queue in Dijkstra’s algorithm, local search is that of RandomOSPF.
the computation for each destination takes O ( ( A ( )= O(lN1) The results for the AT&T WorldNet backbone with different
time, and hence our total time bound is O ( ( N I 2 ) . scalings of the projected demand matrix are presented in Table I.
In each entry we have the normalized cost @* introduced in Sec-
B. The dynamic case tion HI. The normalized cost is followed by the max-utilization
In our local search we want to evaluate the cost of many dif- in parenthesis. For all the OSPF schemes, the normalized cost
ferent weight settings, and these evaluations are a bottleneck for and ma-utilization are calculated for the same weight setting
our computation. To save time, we will try to exploit the fact that and routing. However. for OPT,the optimal normalized cost and
when we evaluate consecutive weight settings, typically only a the optimal max-utilization are computed independently with
few arc weights change. Thus it makes sense to try to be lazy different routing. We do not expect any general routing to be
and not recompute everything from scratch, but to reuse as much able to get the optimal normalized cost and max-utilization si-
as possible. With respect to shortest paths, this idea is already multaneously. The results are also depicted graphically in Fig-
well studied (Ramalingam and Reps [12]), and we can apply ure 5. The first graph shows the normalized cost and the hori-
their algorithm directly. Their basic result is that, for the re- zontal line shows our threshold of 10: for regarding thenetwork
computation, we only spend time proportional to the number of as congested. The second graph shows the max-utilization.
arcs incident to nodes x whose distance dk to t changes. In our The synthetic internetworks were produced using the gener-
experiments there were typically only very few changes, so the ator GT-ITM [14], based on a model of Calvert, Bhattachar-
gain was substantial - in the order of factor 20 for a 100 node jee,Daor, and Zegura [SI, 161. This model places nodes in a
graph. Similar positive experiences with this laziness have been unit square, thus getting a distance 6(x,y) between each pair of
reported in Frigioni et al. [13]. nodes. These distances lead to random distribution of 2-level
The set of changed distances immediately gives us a set of graphs, with arcs divided in two classes: local access arcs and
“update” arcs to be added to or deleted from A t . We will now long distance arcs. Arc capacities were set equal to 200 for local
present a lazy method for finding the changes of loads. We will access arcs and to 1000 for long distance arcs. The above model
operate with a set M of “critical” nodes. Initially, M consists of does not include a model for the demands. We decided to model
all nodes with an incoming or outgoing update arc. We repeat the demands as follows. For each node x , we pick two random
the following until M is empty: First, we take the node y E M numbers O,, Dy E [0,1] . Further, for each pair (z,y) of nodes
which maximizes the updated distance db and remove y from we pick a random number C(,,y! E [0,1]. Now, if theEuclidean
M . Second, set 1 = &(D(y,t) -t C(r,y)EAt Finally, distance (L2) between x and y IS 6 ( x , y), the demand between
x and y is
for each (y, z ) E A t , where At is also updated, if I # ltv,zl,set
l t u , 2 ) = 1 and add z to M . cUOxD,c ( x , y ) e - + ~ y ) ’ ~ A

0-7803-j880-5/00/$I0.00 ( c ) 2000 IEEE 525 IEEE INFOCOM 2000

Here a is a parameter and A is the largest Euciidean distance Following the pattern in our analysis for the normalized cost, we
between any pair of nodes. Above, the 0, and D, model that can ask how much more demand we can deal with before getting
different nodes can be more or less active senders and receivers, max-utiliztion above 1, and again we see that HeurOSPF beats
thus modelling hot spots on the net. Because we are multiplying the oblivious schemes with at least 50%.
three random variables, we have a quite large variation in the The fact that our HeurOSPF provides weight settings and
demands. The factor e-6(rpy)/3A implies that we have relatively routings that are simultaneously good both for our best effort
more demand between close pairs of nodes. The results for the type average cost function, and for the performance guarantee
synthetic networks are presented in Figures 6 9 . type measure of max-utilization indicates that the weights ob-
tained are “universally good” and not just tuned for our partic-
VIII. DISCUSSION ular cost function. Recall that the values for OPT are not for
the same routings, and there may be no general routing getting
First consider the results for the AT&T WorldNet backbone simultaneously closer to the optimal cost and the optimal max-
with projected (non-scaled) demands in Table 5(*). As men- utilization than HeurOSPF. Anyhow, our HeurOSPF is generally
tioned in the introduction, our heuristic, HeurOSPF, is within so close to OPT that there is only limited scope for improve-
2% from optimality. In contrast, the oblivious methods are all ment.
off by at least 15%. In conclusion, our results indicate that in the context of known
Considering the general picture for the normalized costs in demands, a clever weight setting algorithm for OSPF routing is
Figures 5-9, we see that L2OSi’F and RandomOSPF compete a powerful tool for increasing a network’s ability to honor in-
to be worst. Then comes InvCapOSPF and UnitOSPF closely creasing demands, and that OSPF with clever weight setting can
together, with InvCapOSPF being slightly better in all the fig- provide large parts of the potential gains of traffic engineering
ures but Figure 6 . Recall that InvCapOSPF is Cisco’s recom- for supporting demands, even when compared with the possibil-
mendation [3], so it is comforting to see that it is the better of ities of the much more flexible MPLS schemes.
the oblivious heuristics. The clear winner of the OSPF schemes Acknowledgment We would like to thank David Johnson and
is our HeurOSPF, which is, in fact, much closer to the general Jennifer Rexford for some very useful comments. The first au-1
optimum than to the oblivous OSPF schemes.
To quantify the difference between the different schemes,
thor was sponsored by the AT&T Research Prize 1997. i!
note that all curves, except those for RandomOSPF, start off REFERENCES
pretty flat, and then, quite suddenly, start increasing rapidly. [l] D. 0. Awduche, 1. Malcolm, J. Agogbua, M. ODell, and
This pattern is somewhat similar to that in Fig. 1. This is not J. McManus, “Requirements for traffic engineering ove
MPLS,” Network Working Group, Request for Comments!
surprising since Fig. 1 shows the curve for a network consisting ht tp :/ / search.iet f .org/rfc/rfc2702. txt, Septembei
of a single arc. The reason why RandomOSPF does not follow 1999. i
this pattern is that the weight settings are generated randomly [21 J. T. Moy. OSPF: Anatomy of an lnternet Routing Protocal, Addison{
Wesley, 1999.
for each entry. The jumps of the curve for Random in Figure [3] Cisco, Conjguring OSPF, 1997.
8 nicely illustrate the impact of luck in the weight setting. In- [41 E. C. Rosen, A. Viswanathan. and R. Callon. “Mul-
terestingly, for a particular demand, the value of RandomOSPF tiprotocol label switching architecture,” Network
Working Group, Intemet Draft (work in progress).
is the value of the initial solution for our local search heuris- http://search.ietf.org/intemet-drafts/draft-
tic. However, the jumps of RandomOSPF are not transferred to ietf-mpls-arch-06. txt,1999.
HeurOSPF which hence seems oblivious to the quality of the [5] E. W.Zegura, K. L. Calvert, and S . Bhattachajee, “How to model an
intemetwork:’ in Proc. 15th IEEE Conf on Computer Communications
initial solution. (INFOCOM), 1996, pp. 594-602.
Disregarding RandomOSPF, the most interesting comparison [6] K. Calvert, M. Doar. and E. W. Zegura, “Modeling intemet topology:’
IEEE Communications Magazine, vol. 35, no. 6, pp. 160-163, June 1997.
between the different schemes is the amount of demand they can [7] D.L. Woodruff and E. Zemel, “Hashing vectors for tabu search,” Annals
cope with before the network gets too congested. In Section III. ofOperationsResearch, vol. 41, pp. 123-137.1993.
we defined l o $ as the threshold for congestion, but the exact [8] L. G. Khachiyan, “A polynomial time algorithm for linear programming,”
DoW.Akad Nauk SSSR. vol. 244. pp. 1093-1096.1979.
threshold is inconsequential. In our experiments, we see that [9] E. H. L. Aarts and J. K. Lenstra, Eds., Local Search in Combinatorial Op-
HeurOSPF allows us to cope with 50%-110% more demand. timizarion, Discrete Mathematics and Optimization. Wiley-Interscience,
Also, in all but Figure 7, HeurOSPF is less than 2% from be- Chichester, England, June 1997.
[lo] F. Glover, “Future paths for integer programming and links to artificidl
ing able to cope with the same demands as the optimal general intelligence:’ Computers & Operations Research, vol. 13. pp. 533-549,
routing OPT. In Figure 7, HeurOSPF is about 20% from OPT. 1986.
Recall that it is NP-hard even to approximate the optimal cost [l 11 M. Dietzfelbibger, “Universal hashing and k-wise independent random
variables via integer arithmetic without primes,” in Proc. 13th Symp. on
of an OSPF soIution, and we have no idea whether there ex- Theoretical Aspects of Computer Science (STACS), LNCS 1046.1996, pp.
isits OSPF solutions closer to OPT than the ones found by our 22-24, Springer.
[ 121 G. Ramalingam and T.Reps, “An incremental algorithm for a generaliza-
heuristic. tion of the shortest-path problem,” Jounal of Algorithms, vol. 21, no. 2,
If we now turn out attention to the max-utilization, we get pp. 267-305,1996.
the same ordering of the schemes, with InvCapOSPF the winner [I31 D. Frigioni, M. Ioffreda, U. Nanni. and G. Pasqualone. “Experimenth
analysis of dynamic algorithms for the single-source shortest path prob-
among the oblivious schemes and HeurOSPF the overall best lem,” ACM Jounal ofExperimental Algoritlunics. vol 3, article 5.1998.
OSPF scheme. The step-like pattern of HeurOSPF show the [I41 E. W. Zegura, “GT-lTIvf: Georgia tech internetwork topology
impact of the changes in a:. For example, in Figure 5, we see models (software):’ http: / lwww.cc .gatech.edu/ f ac?
Ellen.Zegura/gt-itm/gt-itm.tar.gz.
how HeurOSPF fights to keep the max utilization below 1, in
order to avoid the high penalty for getting load above capacity.

0-7803-S880-5/00/$10.00 (c) 2000 IEEE 526 IEEE INFOCOM 2000

 Demand InvCapOSPF UnitOSPF L2OSPF RandomOSPF HeurOSPF OPT
3709 1.01 (0.15) 1.00 (0.15) 1.13 (0.23) 1.12 (0.35) 1.00 (0.17) 1.00 (0.10)
7417 1.01 (0.30) 1.00 (0.31) 1.15 (0.46) 1.91 (1.05) 1.00 (0.30) 1.00 (0.19)
11126 1.05 (0.45) 1.03 (0.46) 1.21 (0.70) 1.36 (0.66) 1.01 (0.34) 1.01 (0.29)
14835 1.15 (0.60) 1.13 (0.62) 1.42 (0.93) 12.76 (1.15) 1.05 (0.47) 1.04 (0.39)
(*) 18465 1.33 (0.75) 1.31 (0.77) 5.47 (1.16) 59.48 (1.32) 1.16 (0.59) 1.14 (0.48)
22252 1.62 (0.90) 1.59 (0.92) 44.90 (1.39) 86.54 (1.72) 1.32 (0.67) 1.30 (0.58)
25961 2.70 (1.05) 3.09 (1.08) 82.93 (1.62) 178.26 (1.86) 1.49 (0.78) 1.46 (0.68)
29670 17.61 (1.20) 21.78 (1.23) 113.22 (1.86) 207.86 (4.36) 1.67 (0.89) 1.63 (0.77)
33378 55.27 (1.35) 51.15 (1.39) 149.62 (2.09) 406.29 (1.93) 1.98 (1.00) 1.89 (0.87)
37087 106.93 (1.51) 93.85 (1.54) 122.56 (2.32) 476.57 (2.65) 2.44 (1.00) 2.33 (0.97)
40796 175.44 (1.66) 157.00 (1.69) 294.52 (2.55) 658.68 (3.09) 4.08 (1.10) 3.64 (1.06)
44504 246.22 (1.81) 228.30 (1.85) 370.79 (2.78) 715.52 (3.37) 15.86 (1.34) 13.06 (1.16)

TABLE I
AT&T'S PROPOSED BACKBONE WITH 90 NODES AND 274 ARCS AND SCALED PROJECTED DEMANDS, WITH (*) MARKING THE ORIGINAL UNSCALED
DEMAND.

14 1.4 -
12

10
1.2 -
OPT -
: e
U

0
0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000
demand demand

Fig. 5. ATBrT's proposed backbone with 90 nodes and 274 arcs and scaled projected demands.

14 1.4

12 1.2

10 B
.
.I 1
a
-2 0 . 8
U
6

4
z
U
4

o'6
0.4

2 0.2

I 0
0 1000 2000 3000 4000 5000 6000 7000
demand

Fig. 6. 2-level graph with 50 nodes and 148 arcs.

0-7803-5880-5/00/$10.00( c )2000 IEEE 527 IEEE INFOCOM 2000

 14 1.4 -

-
LZOSPF

12
OPT

10

u
:8
6

0
0 1000 2000 3000 4000 5000 6000 0 1000 2000 3000 4000 5000 6000
demand demand

Fig. 7. 2-levelgraph with 50 nodes and 212 arcs.

InvCapOSPF
UnitOSpp
------- 1.4
InvCapOSPF .-. ----
UnitOSPF

-
L20SPP --*--

-
L2OSPF
RandomOSPF --*--- RandomOSPP
HeurOSPP -.-.e-- 1.2 HeurOSPF
OPT OPT

8 1
..-I
9
:.
..-I
4
0.8

i Ob6
0.4

0.2

0
0 1000 2000 3000 4000 5000 6000 1000 2000 3000 4000 5000 6000
demand demand

Fig. 8. 2-level graph with 100 nodes and 280 arcs.

14 1.4
LZOSPF --.A --

-
/!
RandomOSPF
12 ? HeurOSPF --.e-.-
OPT
- 1.2
t
I
10

0.2

0 ' Ll n
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 0 2000 4000 6000 8000 10000 12000 1 4 0 0 0 16000 18000
demand demand

Fig. 9. 2-level graph with 100nodes and 360 arcs.

( c ) 2000 IEEE
0-7803-5880-5/00/$10.00 528 IEEE INFOCOM 2000