From: AAAI-98 Proceedings. Copyright 1998, AAAI (www.aaai.org). All rights reserved.

The Constrainedness Knife-Edge

Toby Walsh
APES Group
Department of Computer Science
University of Strathclyde
Glasgow, Scotland
twcs,strath,ac. uk
Abstract deepens,over-constrained problemstend to become
more constrained, but critically constrained problems
A general rule of thumb is to tackle the hardest from the region inbetween tend to remain critically
part of a search problem first. Manyheuristics constrained. We also observe a simple scaling with
therefore try to branch on the most constrained
problem size for various properties measured during
variable. To test their effectiveness at this, we
measure the constrainedness of a problem dur- search including the ratio of clauses to variables. The
ing search. Werun experiments in several differ- existence of a constrainedness knife-edge helps to ex-
ent domains, using both random and non-random plain the hardness of problems from the phase transi-
problems. In each case, we observe a constrained- tion. It also suggests some new branching heuristics
ness "knlfe-edge" in which critically constrained
problems tend to remain critically constrained. for satisfiability. Similar microscopic studies that look
We show that this knife-edge is predicted by a closely inside search may be useful in other domains.
theoretical lower-bound calculation. Wealso ob-
serve a very simple scaling with problem size Constrainedness within satisfiability
for various properties measured during search in-
cluding the ratio of clauses to variables, and the There has been considerable interest in encoding prob-
average clause size. Finally, we use this picture of
search to propose some branching heuristics for lems into satisfiability and solving them either with
propositional satisfiability. local search procedures like GSAT(Selman, Levesque,
Mitchell 1992) or with the Davis-Putnam proce-
dure (Bayardo ~= Schrag 1997). We therefore began
Introduction our experiments by looking at how the constrained-
hess of satisfiability problems varies during search.
Empirical studies of search procedures usually focus The constrainedness of a satisfiability problem depends
on statistics like run-time or nodes visited. It can on several factors including the clause length (longer
also be productive to use the computer as a "micro-
clauses are less constraining than shorter ones) and
scope", looking closely at the running of the search the number of clauses mentioning a variable (increas-
procedure. To illustrate this approach, we measure ing the number of clauses makes the variable more con-
the constrainedness of problems during search. A gen-
strained). We decided therefore to measure both the
eral purpose heuristic in many domains is to branch on ratio of clauses to variables, and the average clause
the most constrained variable. For example, in graph length during search for the popular random 3-SAT
coloring, the Brelaz heuristic colors a node with the problem class (Mitchell, Selman, ~ Levesque 1992).
fewest available colors, tie-breaking on the number of
We use the Davis-Putnam procedure with unit prop-
uncolored neighbours (Brelaz 1979). Howeffective are
agation but no pure literal deletion. We branch with
heuristics at identifying the most constrained variable?
MOMS heuristic, picking the literal that occurs most
How constrained are the resulting subproblems? To
often in the minimal size clauses. Depth is measured
answer such questions, we measured the constrained-
by the number of assignments. Similar results are
ness of problems during search in several different do-
obtained when depth is measured by the number of
mains using both random and non-random problems.
branch points, and with other heuristics including ran-
We obtained similar results with a wide variety of
dom branching. In each experiment, we simply follow
algorithms and heuristics. In each case, we observed
the heuristic down the first branch, averaging over 1000
a constrainedness "knife-edge". Under-constrained
problems. To reduce variance, we use the same ensem-
problems tend to become less constrained as search
ble of problems in all experiments. We adopt the con-
1 Copyright (c) 1998, AmericanAssociation for Artificial vention that initial parameters are in capitals and that
Intelligence (www.aaai.org). All rights reserved. values measured during search are in lower case.
 4.5 is invariant if depths are scaled linearly with problem
!4
size, N. This simple scaling result mayalso be useful in
3.5 a theoretical analysis of the Davis Putnamprocedure.
3 Other experiments showthat the average clause length
c
2.5 decreases as we decrease the initial ratio of clauses to
2 variables, L/N. Whichof these two factors wins? Does
1.5 the decrease in clause size tighten the constrainedness
1 faster than the decrease in the ratio of clauses to vari-
0.5 ables loosens it? To answer such questions, we need a
i t i i i i
50 Ioo 150 2O0 250 300 35O more precise measure of constrainedness.
depth

3 , , , , , , , , ,
N=500
Figure 1: Ratio of clauses to variables, l/n on the 2.8 N-~00-~---
N=300
-E]-- "
N-~.00-x--
heuristic branch against the depth. 2.6 N=100~-

2.4

In Figure 1, we plot the ratio of clauses to variables 2.2

downthe heuristic branch for random 3-SATproblems 2

from the middle of the phase transition with an initial t.8
clause to variable ratio, L/N = 4.3. As search pro-
1.6
gresses, this ratio drops approximately linearly. How- i i i
o.,o., o.~o14o.~o.o o.,o.~o.,
ever, it drops less rapidly for larger problems. Since d~th/N

not all heuristic branches extend to large depths, there

is some noise at the end of each graph. Other exper- Figure 3: Average clause length, m on the heuristic
iments show that the rate of decay of l/n increases branch against the fractional depth.
as we increase the initial ratio of clauses to variables,
L/N. In Figure 2, we rescale the x-axis linearly with
problem size, N. This rescaling showsthat the gradi- An approximate theory
ent of l/n is inversely proportional to N. Sucha simple (Gent et al. 1996) proposes an approximate theory for
scaling result is very unexpected. It maybe useful in estimating the constrainedness of an ensembleof prob-
a theoretical analysis of the Davis Putnamprocedure. lems. This theory focuses on just two factors: the size
of the problems, and the expected numberof solutions.
Problems which are large but which have a smM1num-
4.5 , , , , , , , , , ber of solutions tend to be over-constrained. On the
4- N=400*~o-
N=3~-o-- other hand, problems which are small but which have a
N=2~ ~--
3.5 - 00 large numberof solutions tend to be under-constrained.
3 Whilst this theory ignores important factors like prob-
2.5 lem structure and symmetries, its predictions are often
2 surprisingly accurate. For instance, the theory predicts
1.5 the location of a phase transition in numberpartition-
N~ 10 -e--
ing with just a 4% error (Gent & Walsh 1996).
0.5 -
If each problemin an ensemblehas a state space with
aep~N
2N states, of which (Sol) are expected to be solutions,
then the constrainedness, g of the ensembleis defined
by,
Figure 2: Ratio of clauses to variables, l/n on the
heuristic branch against the fractional depth.
=def 1 lg2((Sl))
N
As the ratio of clauses to variables drops during This parameter lies within the interval [0,oo). If
search, we might expect that problems become less = 0, problems in the ensemble are completely
constrained. However, the average clause length also under-constrained and every state is a solution. If
decreases as search deepens, tightening the constraints = oo, problems in the ensemble are completely over-
on variables. In Figure 3, we showthat, just like the constrained and no states are solutions. If n < 1, prob-
ratio of clauses to variables, the average clause length lems are under-constrained and are typicMly soluble. If
 > 1, problems are over-constrained and are typically TheconstraJnedness
knife-edge
3
soluble. Aroundn ~ 1, there tends to be a phase tran- L/N=8-~--
12N=7-~--
sition as problems can be both soluble and insoluble. 2.5 UN=6 -Q--
t/N--5 ~--
The hardest problems to solve often occur around such 2 L/N=4-~ -
I./N--~-o---
UN=2 -+---
transitions (Cheeseman,Kanefsky, & Taylor 1991). tJN=l-t~--
1.5
t
Constrainedness knife-edge 1;

Wecan use this definition of constrainedness to de- 0.5

termine whether the decrease in average clause size
0
outweighsthe decrease in the ratio of clauses to vari- 0.1 O.2. O.3 O.4 O.5 O.6 O.7 O.f$ 0.9
deptl~
ables. To estimate n during search, we assume that
the current subproblem is taken from an ensemble in
which problems have the same number of clauses, the Figure 4: The estimated constrainedness, n downthe
same numberof variables, and the same distribution of heuristic branch for random 3-SATproblems with 100
clause lengths. If there are li clauses of length i, then variables and varying initial ratio of clauses to variable.
as each clause of length i rules out the fraction (1 - 21)
of the 2n possible truth assignments, they look similar at every length scMe. At the phase
boundary in computational systems, problems also dis-
<SoO2-.ri(1-i ,, play a form of self-similarity. Branchingdecisions give
i subproblemsthat look neither more or less constrained.
This helps to explain why such problems are difficult
Hence, to solve. Branchingdecisions tell us very little about
1 the problem, giving subproblems that are neither more
~ - E ~.log2(1 - ~) obviously soluble nor more obviously insoluble. We
i will often have to search to a large depth either for a
solution or for a refutation. By comparison, branch-
Note that for a random 3-SATproblem, n is directly
ing on an over-constrMned problem gives a subproblem
proportional to L/N, the ratio of clauses to variables.
that is often even more constrained and hopefully eas-
In Figure 4, we plot the estimated constrainedness
downthe heuristic branch for random 3-SATproblems. ier to show insoluble, whilst branching on an under-
constrained problem gives a subproblem that is ofen
For L/N < 4.3, problems are under-constrained and
even less constrained and hopefully easier to solve.
soluble. As search progresses, ~ decreases as problems
becomemore under-constrained and obviously soluble. Lower bound on constrainedness
For L/N > 4.3, problems are over-constrained and in-
Whenwe branch into a subproblem, the number of so-
soluble. As search progresses, n increases as problems
lutions remaining cannot increase. The expected num-
becomemore over-constrained and obviously insoluble.
At L/N ~ 4.3 problems are on the knife-edge between ber of solutions, (Sol) cannot therefore increase. This
provides a lower boundon g that is a good qualitative
solubility and insolubility. As search progresses, n is
estimate for howthe constrainedness actually varies
roughly constant. Each successive branching decision
gives a subproblem which has the same constrainedness during search. Let xi be the value of g at depth i.
as the original problem, neither more obviously satis- Then,
fiable, nor more obviously unsatisfiable. Only deep in
search does n eventually break one way or the other. ~o = 1 lg2((Sl))
N
As with the ratio of clauses to variables, and the av- Hence,
erage clause length, graphs of the constrainedness dur-
ing search coincide if depths are scaled linearly with log2((Sol)) = N(1- no)
problem size, N. Wehave also observed similar knife-
Thus,
edge behaviour with a random heuristic, and with
an anti-heuristic (that is, one which always branch- lg2((Sl))
> 1
ing against the heuristic) except that values of ~ are - N-i
slightly greater. N(1- no)
= 1
Figure 4 suggests an interesting analogy with sta- N-i
tistical mechanics. At the phase boundaryin physical Nno - i
systems, problems tend to be "self-similar". That is, N-i
Wecan improve this bound slightly by noting that 1979), and Geelens promise heuristic to choose one of
is bounded below by zero. Hence, the m possible colors (Geelen 1992). To estimate
we assume that the graph is drawn from an ensemble
N-~a---i] in which graphs have the same number of nodes, the
ni >
- max(0, N-i" same available colors, and the same numberof edges as
in the current subproblem.If V is the set of uncolored
In Figure 5, we plot this bound on a for random3-SAT
nodes, E is the set of edges between uncolored nodes,
problemswith 100 variables and varying initial ratio of
and mi is the set of colors remaining for node i then
clauses to variable, L/N. Wesee that the behaviour of there are I~iev Imil possible colorings of the nodes, and
n during search observed in Figure 4 is similar to that
each edge (i,j) E rules ou t I~+nm~lthe ofImil.lmjl
predicted by the bound.
pairs of colors between nodes, i and j. Thus,
3
L/N--8 -- N = log2(1- ~ Imil)
D = ~--~log2(lm+
L/N=7.....
2.5 LIN=6 iEV iEV
LIN=5 .....
....
UN--4,3.....
2I L/N---4.....
L/N--,3...... (Sol) ,~ I-I lmd. II (1 Imif3mJl"
I.]N=2......
1.5 IJN=I......

l Hence,
0.5
E(i,j)EE log2(1 - ~)
0
0,2 0.4 0.6 0.8
logdl-+D
dopflVN
In Figure 6, we plot the estimated constrainedness
downthe heuristic branch for a typical register al-
Figure 5: Lower-boundon the constrainedness, n down
location problem. Despite the fact that this plot is
a branch for random3-SATproblems with 100 variables
and varying initial ratio of clauses to variable.
! j
1 i m--40----

f
m--42-+---
111---44
-o--.
Non-random problems 0.8 /i m--46
-. ....
m=49(opt) -~--
The existence of a constrainedness knife-edge helps to m=50 -<~---
m--52-+---
0.6 .: ~.
explain the difficulty of solving randomproblems at m=54-~ --

the phase transition in solubility. Branching decisions

give subproblemswhich are neither more obviously sol-
l Si i
. .~-~--:_
.- . --+o..-~+-~-E~.~l_t3-
-B-O
B-O-~-~-~-[3-O~
m--56--x--
m=60-~-+

uble or insoluble. Weare forced therefore to search

to a large depth either for a solution or for a refu- 0 0.1 0.2 0.3 0.4 0.5
depth/N
tation. Phase transition behaviour has also been ob-
served in problems which are not purely random. For
instance, (Gent &Walsh1995) identifies phase transi- Figure 6: Estimated constrainedness downthe heuris-
tion behaviour in traveling salesperson problems using tic branch for a typical register allocation problemfrom
real geographical data, in graph coloring problems de- the DIMACS library using a forward checking algo-
rived from university examtime-tables, and in Boolean rithm. The probleminstance ("zeroin.i.l.col") has 211
induction and synthesis problem. As a fourth exam- nodes, 4100 edges, and needs m = 49 colors. For
ple, (Gomes & Selman 1997) demonstrate phase tran- m < 48, the estimate for n becomesinfinite before the
sition behaviour in the quasi-group completion prob- end of search as the problembecomesarc inconsistent.
lem. Does the existence of a constrainedness knife-edge
help to explain the difficulty of solving problemsat the for a single problem instance, we observe a "knife-
phase boundary in such non-random problems? edge". With less than 49 colors, the problem is
To answer this question, we ran some experiments over-constrained and insoluble. As search progresses,
with graph coloring problems from the DIMACS the constrainedness increases rapidly. Each branch-
benchmark library. Weused the register allocation ing decision results in a subproblemthat is more ob-
problems as these are based on real code. To color the viously insoluble. With more than 49 colors, the
graphs, we use a forward checking algorithm with the problem is under-constrained and soluble. As search
Brelaz heuristic to pick the next node to color (Brelaz progresses, the constrainedness only increases slightly.
Each branching decision gives a subproblemthat is of
similar constrainedness and difficulty to solve. Simi- Iog2(L)/N=2,0
Iog2(L)/N=1.8
-~
/,/: ~ ! ~:
Iog2(L)/N=l.6
-IJ-- /~,,
I :: -~~; :!
lar behaviour is seen with the other register allocation Io92(LyN=l.4-x-- ~/j~ ~ i !
Iog2(L)/N=l.2~-- //.." i [ j i;
problems in the DIMACS library. IOg2IL)/N-10
.... ~/Jd,: i-:

Constrainedness within optimization /,.ZiL;i!

Phase transition behaviouris not restricted to decision ,
problemslike propositional satisfiability. Certain op-
timization problems like numberpartitioning and the
traveling salesperson problem also exhibit phase tran- 0 0.1 0.2 0.3 0.4 0.5 0.6
d~tigN
0.7 0.8 0.9

sitions (Gent & Walsh 1996; Zhang & Korf 1996). (A)
we observe a constrainedness knife-edge when solving 10 .... ,;:i
such optimization problems? I Iog2(L)/N-~.0 I[~
/ Iog2(L)/N=1.8 ...... Jj ~
[,, :" J
To explore this question, we ran some experiments 8 [-I Iog2(L)/N=1.4
Iog2(t)/N=i.6
.....
/] :, [
/ Iog2ll.yl~-l.2.... //"!i
with the CKKoptimization procedure for number par- / ~(L~=I.0
-- /i ;
o F ~(L~4~--0.6...... / ! :: - f
titioning (Korf 1995). Given a bag of N number, i ~2(L~--0.4......
- / i : ] !
wish to find a partition into two bags that minimizesA, 4/ //.../
.."..."/i
the difference between the sum of the two bag. (Gent
& Walsh 1996) shows that for partitioning n numbers . ...........
drawn uniformly at random from(0,1], n ~ loa~(1)/n.
To estimate n during search, we assume that the num- 0 0.2 0.4 0.6 0.8
deptWN
bers left are taken from such an ensemble and that
their size, l is twice the sample average. In Figure 7, (B)
we plot this estimate for the constrainedness during
search. For comparison, we also plot the lower bound Figure 7: The constrainedness, n downthe heuristic
on n using the same scales. Weagain observe a con- branch for number partitioning problems with N =
strainedness knife-edge. Althoughthere is not a tran- 30 numbers, and varying L. (A) estimated n. (B)
sition between soluble and insoluble problems (since theoretical lower-boundto same scale.
there is always an optimal partition), there is now
transition between optimization problems with perfect
partitions (that is, in whichA < 1) and those without, that this heuristic is effective on hard and unsatis-
fiable random 3-SATproblems. For instance, for 50
and verifying the optimality of a partition with A > 1
can be costly. variable unsatisfiable problems at L/N = 4.3, the me-
dian nodes searched using this heuristic is 2,575 com-
Constrainedness as a heuristic pared to 3,331 nodes for MOMS heuristic, and 7,419
nodes for the heuristic that minimizes n. On the other
Knowledgeabout the existence of a constrainedness hand, maximizing~ is less effective on hard and sat-
knife-edge may help us design more effective search isfiable problems. For 50 variable satisfiable problems
procedures. For instance, for soluble problems, it sug- at L/N = 4.3, the median nodes searched when maxi-
gests that we should try to get off the knife-edge as mizing n is 1,487 compared to 164 nodes with MOMs
quickly as possible by branching into the subproblem heuristic, and 104 nodes with the heuristic that mini-
that is as under-constrained as possible. That is, as mizes n. An adaptive heuristic that switches between
suggested in (Gent et al. 1996), we should branch into minimizing and maximizing n depending on an esti-
the subproblemthat minimizes n. To test this thesis, mate of the solubility of the problem may therefore
we implemented a branching heuristic for the Davis- offer good performance.
Putnam procedure that branches on the literal which
gives the subproblem with smallest n. In Table 1, we
showthat this heuristic performs well on hard and sat- Related work
isfiable random 3-SATproblems. Most theoretical studies of the Davis-Putnam pro-
For insoluble problems, the existence of a con- cedure have used the easier constant probability
strainedness knife-edge suggests that we should branch model. One notable exception is (Yugami1995) which
into the sub-problemthat is as over-constrained as pos- computes the average-case complexity of the Davis-
sible. That is, we should branch into the subprob- Putnamprocedure for the random3-SATproblem class.
lem that maximizes n. Initial experiments suggest Freemanhas studied experimentally the running of the
 N MOM KAPPA Acknowledgments
25 11 1 I wish to thank membersof the APESgroup, especially
50 164 104 Barbara Smith and Inn Gent, for their commentsand
75 1129 580 criticisms. The author is supported by EPSI~Caward
100 3903 1174 GR/K/65706.

References
Table 1: Median nodes searched by the Davis-Putnam
procedure for satisfiable random 3-SATproblems at Bayardo, R., and Schrag, 1%. 1997. Using CSPlook-
back techniques to solve real-world SATinstances. In
L/N = 4.3, branching either with MOMsheuristic,
Proceedings of the 14th National Conference on AL
or to minimize the constrainedness (KAPPA).
Brelaz, D. 1979. Newmethods to color the vertices
of a graph. Commincations of ACM22:251-256.
Davis-Putnam procedure on random 3-SAT problems Cheeseman, P.; Kanefsky, B.; and Taylor, W. 1991.
(Freeman1996). Unlike here, where the focus is on the Wherethe really hard problems are. In Proceedings
heuristic branch, Freemancomputesaverages across all of the 12th IJCAI, 331-337.
branches in the search tree. He identifies an "unit cas- Freeman, J. 1996. Hard random 3-SATproblems and
cade", a depth in the search tree where unit propaga- the Davis-Putnamprocedure. Artificial Intelligence
tion greatly simplifies the problem. The ineffectiveness 81(1-2):183-198.
of unit propagation above this depth helps to explain Geelen, P. 1992. Dual viewpoint heuristics for binary
the hardness of problems at the phase transition. constraint satisfaction problems.In Proceedingsof the
Gent and Walsh have studied experimentally the lOth ECAI, 31-35.
running of local search procedures for satisfiability Gent, I., and Walsh, T. 1993. An empirical analysis
(Gent ~ Walsh 1993). They show that various proper- of search in GSAT.Journal of Artificial Intelligence
ties like the percentage of clauses satisfied, and the Research 1:23-57.
numberof variables offered to flip are invariant if Gent, I., and Walsh, T. 1995. Phase transitions from
depths are scaled linearly with problem size. This mir- real computational problems. In Proceedings of the
rors the result here on the scaling of the constrained- 8th Int. Symp.on Artificial Intelligence, 356-364.
ness, the ratio of clauses to variables and the average Gent, I., and Walsh, T. 1996. Phase transitions
clause size. Such simple scaling results maybe useful and annealed theories: Numberpartitioning as a case
in the theoretical analysis of these search procedures. study. In Proceedings of ECAI-96.
Gent, I.; MacIntyre, E.; Prosser, P.; and Walsh, T.
Conclusions 1996. The constrainedness of search. In Proceedings
of the 13th National Conference on AL
Wehave measured how the constrainedness of prob- Gomes, C., and Selman, B. 1997. Problem structure
lems varies during search in several different prob- in the presence of perturbations. In Proceedingsof the
lem domains: both decision problems like proposi- l~th National Conference on AI, 221-226.
tional satisfiability and graph coloring, and optimiza- Korf, R. 1995. From approximate to optimal solu-
tion problems like number partitioning. Our experi- tions: A case study of number partitioning. In Pro-
ments have used both random and non-random prob- ceedings of the 14th IJCAL
lems. In each case, we observed a constrainedness Mitchell, D.; Selman, B.; and Levesque, H. 1992.
"knife-edge" in which critically constrained problems Hard and Easy Distributions of SATProblems. In
tended to remain critically constrained. The existence Proceedings of the lOth National Conference on AI,
of a constrainedness knife-edge helps to explain the 459-465.
hardness of problems from the phase transition. We Selman, B.; Levesque, H.; and Mitchell, D. 1992. A
have shown that a lower-bound calculation predicts NewMethodfor Solving Hard Satisfiability Problems.
this knife-edge theoretically. Wehave also observed In Proceedingsof the 10th National Conference on AI,
a very simple scaling with problem size for various 440-446.
properties measuredduring search like the constrained- Yugami, N. 1995. Theoretical analysis of Davis-
ness, the ratio of clauses to variables, and the average Putnamprocedure and propositional satisfiability. In
clause size. Finally, we have used the existence of a Proceedings of the 14th IJCAI, 282-288.
constrainedness knife-edge to propose somebranching Zhang, W., and Korf, R. 1996. A study of complex-
heuristics for propositional satisfiability. Weconjec- ity transitions on the asymmetictraveling salesman
ture that similar microscopicstudies that look closely problem. Artificial Intelligence 81(1-2):223-239.
inside search maybe useful in other domains.