Flags and Lattice Basis Reduction

Flags and Lattice Basis Reduction Hendrik W. Lenstra, Jr. Abstract. In this lecture we give a self-contained introduction to the theory of lattices in Euclidean vector spaces. We reinterpret a large class of lattice basis reduction algorithms by using the concept of a “flag”. In our reformulation, lattice basis reduction algorithms are more appropriately called “flag reduction” algorithms. We address a problem that arises when one attempts to find a particularly good flag for a given lattice. 1. Introduction A lattice is a discrete subgroup of a Euclidean vector space. Every lattice has a basis, and a lattice basis reduction algorithm is an algorithm that transforms a given basis for a given lattice into a basis consisting of relatively short vectors. The present lecture is devoted to a conceptual discussion of an important class of lattice basis reduction algorithms. This class includes an algorithm that I introduced in 1981 [4], its close relative that is known as the LLL or Lovász basis reduction algorithm from 1982 [3], and many variants of these algorithms that were proposed in subsequent years. The original applications of basis reduction algorithms to integer programming ([5, 1]) and to algorithmic number theory ([3, 2]) exemplify the scope of their importance in pure and applied mathematics. The central notion in the present discussion is that of a flag for a lattice. A flag carries a little less information than a basis; the definition is given in section 6. The basis reduction algorithms that I consider can be reinterpreted in terms of flags; one may say that they transform a given flag for a lattice into a ‘reduced’ flag for the same lattice. To this end they perform a series of successive steps, each step replacing a flag by a ‘neighbouring’ one that is closer to being reduced. We picture this procedure by means of a directed graph, of which the vertices represent all flags for a given lattice, and the arcs the steps that are permitted. For the algorithm to be efficient, it is necessary that not too many steps are performed in succession. This leads to the problem of giving an upper bound for the length of a directed path in the graph that starts from a given vertex. In section 8 I present such an upper bound. It may be considered satisfactory if one considers only lattices of fixed rank. It is an interesting open problem to find an upper bound that has a better behaviour as a function of the rank. 2 H. W. Lenstra, Jr. 2. Euclidean Vector Spaces A Euclidean vector space is a finite dimensional vector space E over the field R of real numbers equipped with a map
,  : E × E → R satisfying
w + x, y =
w, y +
x, y,
rx, y = r
x, y,
x, y =
y, x,
z, z > 0 for all r ∈ R and w, x, y, z ∈ E, z = 0. We refer to the map
,  as the inner product on E. Any Euclidean vector space E is a metric space with distance function d : E ×E → R defined by d(x, y) =
x−y, x−y1/2 . For each non-negative integer n, the vector space Rn is a Euclidean n vector space with the standard inner product defined by
(xi )ni=1 , (yi )ni=1  = i=1 xi yi . Let E be a Euclidean vector space, and let D ⊂ E be a subspace. Then the restriction of
,  to D × D makes D into a Euclidean vector space. Let D† = Hom(D, R) be the dual of D, and write D⊥ for the kernel of the linear map E → D† sending x ∈ E to the map y →
x, y. One has D ∩ D⊥ = 0, so the natural maps D → E/D⊥ → D† are injective, and by dim D = dim D† they are isomorphisms. It follows that each w ∈ E has a unique representation as w = x+y, with x ∈ D and y ∈ D⊥ ; this is the orthogonal decomposition of w with respect to D. The quotient space E/D becomes a Euclidean vector space as well, since it is canonically isomorphic to the subspace D⊥ of E. One also concludes that E is canonically isomorphic to E † . Applying the above to one-dimensional subspaces, one easily proves by induction on dim E that for every Euclidean vector space E there is a linear isomorphism from the standard Euclidean vector space Rdim E to E that preserves inner products. For example, if one makes the field C of complex numbers into a Euclidean vector space by putting
x, y = (xȳ + yx̄)/2 (= the real part of xȳ), then the isomorphism R2 → C sending (a, b) to a + bi preserves inner products. Proposition 2.1. Let D be a finite dimensional vector space over R, and let
,  : D × D → R be a map satisfying
w + x, y =
w, y +
x, y,
rx, y = r
x, y,
x, y =
y, x, and
x, x ≥ 0 for all r ∈ R, w, x, y ∈ D. Write rad D = {x ∈ D :
x, x = 0}. Then rad D is a subspace of D, and if we write E = D/ rad D then
,  can be written as the composition of the natural map D × D → E × E and a uniquely determined map E × E → R; moreover, the latter map makes E into a Euclidean vector space. Proof. Let w ∈ rad D, y ∈ D. Then one has 2r
w, y+
y, y =
rw +y, rw +y ≥ 0 for every r ∈ R, and therefore
w, y = 0. One readily deduces that rad D is a subspace and that for any x, x′ , y, y ′ ∈ D with x ≡ x′ mod rad D and y ≡ y ′ mod rad D one has
x, y =
x′ , y ′ . Hence
,  factors through E × E. The last statement is now immediate.
Flags and Lattice Basis Reduction 3 3. Lattices in a Euclidean Vector Space Let E be a Euclidean vector space. A lattice in E is an additive subgroup L of E for which there exists a positive real number l such that every z ∈ L, z = 0, satisfies
z, z ≥ l; equivalently, it is an additive subgroup of E that is discrete in the topology induced by the metric on E. Proposition 3.1. Let L be a lattice in a Euclidean vector space E, and let r, l ∈ R be real numbers with r ≥ 0, l > 0 such that every z ∈ L, z = 0, satisfies
z, z ≥ l. Write n for the dimension of the subspace of E spanned by L. Then we have  #{x ∈ L :
x, x ≤ r} ≤ (1 + 2 r/l)n . Proof. Replacing E by the subspace spanned by L we may assume n = dim E. Since L is an additive subgroup, any two distinct√ elements of L differ by a non-zero element of L and therefore have distance at least l. Hence the open n-dimensional √ balls balls with radius l/2 centered at all x ∈ L are pairwise disjoint. All of these √ r+ whose center x satisfies
x, x ≤ r are contained in the open ball with radius √ l/2 centered at 0. Computing volumes we find √ √ √ #{x ∈ L :
x, x ≤ r} · ( l/2)n ≤ ( r + l/2)n . (It is practical to rescale the volume on E so that the unit ball has volume 1.)
Let Z denote the ring of integers. Proposition 3.2. Let E be a Euclidean vector space, let b1 , b2 , . . . , bn ∈ E be linearly independent, and for each ilet bi = (bi − b∗i ) + b∗i be the orthogonal ∗ decomposition of bi with respect to 0 for j < i, and j<i Rbj ; so
bi , bj  =   ∗ ∗ bi − bi ∈ j<i Rbj . Then all bi are non-zero, and for each z ∈ i Zbi , z = 0, one has
z, z ≥ mini
b∗i , b∗i . Remark. If the bi are not supposed to be linearly independent, then the dimension of the subspace they span equals the number of i for which b∗i = 0.    Proof. We have b∗i = 0 since bi ∈ / j<i Rbj . For z = i ni bi ∈ i Zbi , z = 0, ∗ ∗ choose i maximal with ni = 0. Then  z = (z − ni bi ) + n2 i bi∗ is∗ the orthogonal decomposition of z with respect to j<i Rbj , so
z, z ≥ ni
bi , bi  ≥
b∗i , b∗i .
Proposition 3.3. Let E be a Euclidean vector space and let L be a subset of E. Then L is a lattice in E if and only if there is a linearly independent subset B ⊂ E with L = b∈B Zb. If B and L are as in proposition 3.3, then B is called a basis for L. Its cardinality #B equals the number n defined in proposition 3.1, so it depends only on L; it is called the rank rk L of L. One has rk L ≤ dim E. Proof. The if-part follows from proposition 3.2. For the only if-part, let C ⊂ L be a basis for the subspace D of E spanned by L. Each w ∈ L can be written 4 H. W. Lenstra, Jr.  as w = x + ywith x ∈ M = c∈C Zc and y in the intersection of L with the bounded set c∈C [0, 1)c. By proposition 3.1, that intersection is finite, so M has finite index m (say) in L. Lagrange’s theorem from group theory now implies that mL ⊂ M , so L is a subgroup of finite index of the free abelian group m−1 M . Therefore L has a basis B over Z with #B = #C, and B is linearly independent since its span contains C.
4. Lattices We next define lattices in an absolute sense, without reference to a Euclidean vector space. A lattice is a finitely generated abelian group L equipped with a map q : L → R satisfying the following three conditions: (i) q(x + y) + q(x − y) = 2q(x) + 2q(y) for all x, y ∈ L (the parallelogram law); (ii) q(z) = 0 for all z ∈ L, z = 0; (iii) for each real number r, the set {x ∈ L : q(x) ≤ r} is finite. An isomorphism from a lattice L, q to a lattice L′ , q ′ is a group isomorphism f : L → L′ such that for all x ∈ L one has q(x) = q ′ (f (x)); if such a map exists then the lattices L and L′ are called isomorphic. By proposition 3.1, any lattice in a Euclidean vector space becomes a lattice in the sense just defined if we put q(x) =
x, x. We prove that, up to isomorphism, any lattice can be obtained in this way. Proposition 4.1. Any lattice is isomorphic to a lattice in a Euclidean vector space.   Proof. Let L, q be a lattice. For x, y ∈ L, define
x, y = q(x + y) − q(x − y) /4. The parallelogram law implies q(x − y) = q(y − x), so we have
x, y =
y, x. Let w, x, y ∈ L. We have by the parallelogram law q(w + x + y) + q(w − x + y) = 2q(w + y) + 2q(x) , q(w + x − y) + q(w − x − y) = 2q(w − y) + 2q(x) , q(w + x + y) + q(w − x − y) = 2q(x + y) + 2q(w) , q(w + x − y) + q(w − x + y) = 2q(x − y) + 2q(w) . Taking the alternating sum and dividing by 8 we find that
w + x, y =
w, y +
x, y. One readily checks that q(0) = 0 and
x, x = q(2x)/4 = q(x), for x ∈ L. If x ∈ L satisfies q(x) < 0, then one has q(mx) = m2 q(x) < 0 for all non-zero m ∈ Z, so x has infinite order, and one obtains a contradiction with (iii); hence q(x) ≥ 0 for all x ∈ L. Write D = R ⊗Z L, and let
,  : D × D → R be the R-bilinear function induced by
,  : L × L → R. For each positive integer m and each x ∈ L the element z = (1/m) ⊗ x of D satisfies
z, z = q(x)/m2 ≥ 0, and since the set of all z ∈ D of this form is dense in D one has
z, z ≥ 0 for all z ∈ D. From proposition 2.1 one now obtains a Euclidean vector space E = D/ rad D such that the group homomorphism f : L → E sending x to the coset (1 ⊗ x) mod rad D satisfies q(x) =
f (x), f (x). By property (ii) the map is injective, and using (iii) Flags and Lattice Basis Reduction 5 one deduces that f (L) is a lattice in E that is isomorphic to L. (Comparing ranks and dimensions one also finds rad D = 0, so D = E.)
  Remark. In the sequel, we shall write
x, y = q(x + y) − q(x − y) /4 for x, y in a lattice, and lattices may tacitly be assumed to be embedded in a Euclidean vector space. This is justified by proposition 4.1 and its proof. Rank and determinant Two important numerical invariants attached to any lattice L are its rank rk L and its determinant d(L). The rank is the unique nonnegative integer n for which there is an isomorphism L ∼ = Zn of abelian groups. The determinant is defined by  1/2 d(L) = det
b, b′  b,b′ ∈B , where B is a basis of L; if L is a lattice in Rn with basis equal to the set of columns of a non-singular n × n matrix B, then one has d(L) = |det B|. One way to prove that d(L) is well-defined is by showing the limit relation lim r→∞ #{x ∈ L : q(x) ≤ r} = 1, ωn rn /d(L) which is valid for any lattice L of rank n. Here we write ωn = π n/2 / n2 ! for the standard volume of the unit ball in Rn ; the factor n2 ! = Γ(1+ n2 ) may be computed √ from 0! = 1, 21 ! = π/2, and z! = z · (z − 1)!. We have d(L) = 1 if rk L = 0. Proposition 4.2. Let L, q be a lattice of positive rank n. Then there exists x ∈ L with x = 0 and q(x) ≤ n · d(L)2/n . Proof. Assume that L is a lattice in Rn , and write vol for the standard n-di [0, 1)b. mensional volume. Let B ⊂ Rn be a basis for L, and write F = b∈B Then vol F = d(L), and Rn is the disjoint  union of the sets x + F , x ∈ L. Let l = min{q(x) : x ∈ L, x = 0}, and write t = l/n, so that the assertion of propon n sition 4.2 is equivalent to tn ≤ d(L). Let C be the √ cube [0, t) in R . Any two √ elements of C have distance smaller than t n = l, so their difference is not a non-zero element of L. Hence the sets −x + C, x ∈ L, are pairwise disjoint. Since C is the disjoint union of the sets (x + F ) ∩ C, x ∈ L, we conclude that       tn = vol C = vol F ∩ (−x + C) vol (x + F ) ∩ C = x∈L x∈L    = vol F ∩ x∈L (−x + C) ≤ vol(F ) = d(L) , as required.
√ Remark. Replacing the cube in the proof by an open ball of radius l/2 one −2/n finds the better inequality q(x) ≤ 4ωn · d(L)2/n , with ωn as above, and further −2/n improvements are possible. One has 4ωn = 2n/(πe + o(1)) for n → ∞. 6 H. W. Lenstra, Jr. Sublattices and quotient lattices Let L, q be a lattice, and let K be a subgroup of L. Then the restriction of q to K makes K into a lattice, a sublattice of L. We next restrict to pure subgroups. In general, a subgroup K of an additively written abelian group L is called pure if for all positive integers m one has mK = K ∩ mL. If L is a lattice, this property is equivalent to L/K being torsion-free; and if L is a lattice in a Euclidean vector space E, then it is equivalent to the existence of a subspace D of E such that K = L ∩ D, and also to L having a basis that contains a basis for K. Now suppose that K is a pure sublattice of a lattice L. Then the map q ′ : L/K → R defined by q ′ (x + K) = inf{q(mx − y)/m2 : m ∈ Z, m = 0, y ∈ K} makes L/K into a lattice. To prove this, one embeds L as a lattice in a Euclidean vector space E, one defines D to be the subspace of E spanned by K, and one verifies that q ′ is induced by the inclusion of L/K in the Euclidean vector space E/D. One has rk K + rk(L/K) = rk L, d(K) · d(L/K) = d(L) . Proposition 4.3. Let L be a lattice and let r be a real number. Then the number of sublattices K of L with d(K) ≤ r is finite. Proof. For any subgroup K ⊂ L, with R-linear span R · K, the subgroup K ′ = L ∩ (R · K) is pure, the number m = index[K ′ : K] is finite, and one has d(K) = m · d(K ′ ). Hence we may restrict to pure subgroups. We apply induction on rk L. The set of non-zero b in L with q(b) ≤ max{i · r2/i : 1 ≤ i ≤ n} is finite, and by proposition 4.2 any non-zero subgroup K ⊂ L with d(K) ≤ r contains at least one of them. If K is a pure subgroup containing a given such b, then it also contains the pure subgroup Lb = L ∩ Rb, and K/Lb is a pure subgroup of L/Lb with d(K/Lb ) = d(K)/d(Lb ). Now apply the induction hypothesis to each L/Lb .
Remark. An alternative proof of proposition 4.3 makes use of exterior powers. For subgroups of rank 1, one uses defining property (iii) of lattices. Generally, if K ⊂ L is a subgroup of rank i, then ∧i K ⊂ ∧i L is a subgroup of rank 1, and ∧i L has a natural lattice structure for which d(∧i K) = d(K); in addition, K is ‘almost’ determined by ∧i K in the sense that another subgroup J ⊂ L of rank i satisfies ∧i J = ∧i K if and only if J is a subgroup of L ∩ (R · K) of the same index as K. Remark. It follows from proposition 4.3 that there is a positive lower bound for the determinants of the subgroups of a given lattice. Explicitly, any subgroup K ⊂ L  i/2 with rk K = i > 0 satisfies d(K) ≥ min{q(x) : x ∈ L, x = 0}/i , by proposition 4.2. The dual Let L be a lattice in a Euclidean vector space E with dim E = rk L. Then L† = {x ∈ E :
x, L ⊂ Z} is also a lattice in E, the dual (or polar ) of L. One has rk L† = rk L, d(L† ) = d(L)−1 , L†† = L . Flags and Lattice Basis Reduction 7 If L is a lattice in Rn with basis equal to the set of columns of a certain nonsingular matrix, then the columns of the inverse transpose matrix form a basis for L† . If desired, one can also define the dual without reference to a Euclidean vector space, by taking L† = Hom(L, Z) and letting q(f ), for f ∈ L† , be the infimum of all non-negative real numbers r with the property that for all x ∈ L one has f (x)2 ≤ r · q(x). Let L be a lattice, with dual L† , and let K ⊂ L be a pure sublattice. Then ⊥ K = {x ∈ L† :
x, K = 0} is a pure sublattice of L† that may be identified with (L/K)† , and K † may be identified with L† /K ⊥ ; in addition, one has K ⊥⊥ = K. 5. Algorithmic Problems In the present section we discuss a few fundamental and frequently encountered problems concerning lattices. The first is the homogeneous approximation problem: given a non-zero lattice L, find a non-zero element x ∈ L with q(x) smallest possible. The informal formulation allows many interpretations. For example, the lattice may be ‘given’ in some theoretical sense, and ‘finding’ x may be meant purely existentially, so that proposition 4.2 goes some way towards solving the problem. We are mainly interested in an algorithmic interpretation, in which the lattice is ‘given’ in some numerical manner, and likewise its elements have a numerical representation; the problem of ‘finding’ x is then to be interpreted algorithmically, and one wants not just q(x) but also the run time of the algorithm to be small. One will have to allow for a trade-off between the latter two quantities, and the requirement that q(x) be ‘smallest possible’ may be taken to mean: smallest possible given the time that one is willing to spend. One way of specifying a lattice L numerically is by means of a real m × n matrix B of rank n; then L is embedded in the Euclidean vector space Rm , the columns of B forming a basis, and an element x ∈ L is either represented as a real m-vector or as an integral n-vector consisting of the coefficients of x on that basis. In order to avoid rounding problems one may require the entries of B to be rational. A second way of specifying L is by means of a real positive definite symmetric n×n matrix A; in this case L is the group Zn , its elements are represented as integral n-vectors, and
x, y = xT Ay for x, y ∈ L. Again one may require the entries of A to be rational. One easily transforms the first type of representation into the second by taking A = BT · B, and this transformation preserves rationality. One can also transform the second representation into the first, but complications arise if one wishes to do this by means of a polynomial time algorithm that preserves rationality and keeps m low. There are other possibilities of representing lattices numerically, but the two that we just mentioned appear to be the most convenient ones for algorithmic purposes. Of the many algorithmic situations giving rise to the homogeneous approximation problem we mention a single one; namely, the problem of factoring a given one-variable polynomial f with rational coefficients into irreducible factors, which 8 H. W. Lenstra, Jr. was considered in [3]. In this case, one can take the lattice to consist of integer polynomials of a certain degree that assume a very small value in a suitably constructed p-adic zero of f , and one proves that any sufficiently short non-zero vector in that lattice must be an irreducible factor of f . The homogeneous approximation problem has also appeared under the following guise: given a lattice L in a Euclidean vector space E of dimension rk L, find x ∈ E with L ⊂ (Rx)⊥ + Zx and
x, x largest possible. Geometrically, this amounts to asking for a hyperplane H in E such that L is contained in the union of a collection of maximally widely spaced translates of H; namely, take H = (Rx)⊥ and consider translates with successive distances equal to
x, x1/2 . Such a hyperplane is useful when one wishes to enumerate elements of L that lie in a certain bounded region, which occurs in the context of integer programming (see [5]). A given non-zero vector x ∈ E satisfies L ⊂ (Rx)⊥ + Zx if and only if x/
x, x belongs to the dual L† of L, so the problem is equivalent to the homogeneous approximation problem for L† . Finally, one frequently encounters the inhomogeneous approximation problem: given a lattice L in a Euclidean vector space E, and x ∈ E, find y ∈ E with x−y ∈ L and
y, y smallest possible. In other words, one wishes to ‘round’ a given element x of E to an element w of L such that the ‘error’ d(x, w) is minimal. It is a mistake to think that the special case x = 0 of the inhomogeneous approximation problem amounts to the homogeneous approximation problem (since one takes w = y = 0); but it is true that solving 2rk L − 1 inhomogeneous approximation problems suffices to solve the homogeneous approximation problem; namely, let x range over coset representatives of all non-trivial elements of 21 L/L. All problems that we mentioned can to a certain extent be solved if a reduced basis of the lattice is available. The notion of a ‘reduced basis’ has many different definitions, and one usually chooses the most convenient one for the purpose at hand. Different definitions are rarely logically equivalent, but typically bases that are reduced in different senses share many qualitative properties: they consist of ‘fairly short’ vectors that stand at ‘almost right’ angles, the product of their lengths is a ‘fair’ approximation to the determinant of the lattice, and, of course, they yield solutions to the three problems formulated above. In the next section we shall consider flags of a lattice. The notion of a flag is a little weaker than the notion of a basis, but it still carries enough information to assist us in solving our three problems. Finding a reduced basis for a given lattice is done by means of a lattice basis reduction algorithm, which replaces a given basis for a given lattice by a reduced basis for the same lattice. We shall not present any of these. Instead, we describe in very general terms a flag reduction algorithm, that is, a procedure that replaces a given flag of a given lattice by what might be called a ‘reduced flag’ of the same lattice; but we refrain from giving a rigorous definition of the latter term. Many existing lattice basis reduction algorithms, including those presented in [4] and [3], may be interpreted as flag reduction algorithms, and fit as such under our general description. Flags and Lattice Basis Reduction 9 6. Flags Let L be a lattice, and write n = rk L. A flag of L is a sequence F = (Fi )ni=0 of pure sublattices Fi of L satisfying rk Fi = i (for 0 ≤ i ≤ n) and Fi−1 ⊂ Fi (for 0 < i ≤ n); clearly wemust have F0 = {0} and Fn = L. Every basis (bi )ni=1  n of L gives rise to the flag j≤i Zbj i=0 , and one readily checks that every flag of L is of this form. In order to express when two bases (bi )ni=1 and (ai )ni=1 of L give rise to the same flag, let (b∗i )ni=1 be defined as in proposition 3.2, and (a∗i )ni=1 analogously. Then the two bases give rise to the same flag of L if and only if for each i one has b∗i = ±a∗i ; or, equivalently, if and only if there are integers cij , for i 1 ≤ j ≤ i ≤ n, with bi = j=1 cij aj and cii = ±1 for all i. In an algorithmic context one may wish to represent a flag numerically. Assuming L and its elements to be represented in one of the manners described n in section 5,n one can do this by specifying a basis (bi )i=1 of L; then the flag is j≤i Zbj i=0 . As we just noted, certain changes in the basis do not change the flag. This freedom is often used in order to achieve that the real numbers µij for i−1 which bi − b∗i = j=1 µij b∗j satisfy |µij | ≤ 12 . Let F = (Fi )ni=0 be a flag of L. The size s(F) of F is defined by n  s(F) = d(Fi ) . i=0 For 1 ≤ i ≤ n, the i-th successive distance li (F) is defined by li (F) = d(Fi /Fi−1 ); if F is obtained from a basis (bj )nj=1 , and (b∗j )nj=1 is as above, then one has li (F) = i n
b∗i , b∗i 1/2 . One has d(Fi ) = j=1 lj (F) for 0 ≤ i ≤ n, and s(F) = j=1 lj (F)n+1−j . It is an easy consequence of proposition 4.3 that L has, for any real number r, only finitely many flags F with s(F) ≤ r. ⊥ Let again F = (Fi )ni=0 be a flag of L. Then F⊥ = (Fn−i )ni=0 is a flag of L† , the flag dual to F. One has li (F⊥ ) = ln+1−i (F)−1 , s(F⊥ ) = s(F)/d(L)n+1 , F⊥⊥ = F for 1 ≤ i ≤ n. We shall in particular be interested in flags F with the property that li+1 (F) is not much smaller than li (F), for each i = 1, . . . , n − 1. The following result and its proof show the relevance of such flags for the homogeneous approximation problem formulated in the previous section. Proposition 6.1. Let c be a real number with c ≥ 1, let L be a non-zero lattice in a Euclidean vector space E of dimension n = rk L, and let F be a flag of L with the property li+1 (F)2 ≥ c−1 · li (F)2 for 0 < i < n. Then we have c1−n · l1 (F)2 ≤ min{q(x) : x ∈ L, x = 0} ≤ l1 (F)2 ln (F)2 ≤ max{
x, x : x ∈ E, L ⊂ (Rx)⊥ + Zx} ≤ cn−1 · ln (F)2 . n  ∗ n Proof. Let (bi )ni=1 be a basis of L such that F = j≤i Zbj i=0 , and let (bi )i=1 be as in proposition 3.2. By proposition 3.2, we have min{q(x) : x ∈ L, x = 0} ≥ 10 H. W. Lenstra, Jr. mini
b∗i , b∗i  = mini li (F)2 . The hypotheses imply that li (F)2 ≥ c1−n · l1 (F)2 , and the first inequality follows. The second inequality follows from l1 (F)2 = q(b1 ). One proves the last two inequalities by applying the first two to the dual flag. Note that x = b∗n satisfies L ⊂ (R · x)⊥ + Zx and ln (F)2 =
x, x.
Remark. The flags considered in proposition 6.1 also yield a fairly good solution to the inhomogeneous approximation problem. Namely, if the notation and hypotheses are as in proposition 6.1 and its proof, then for every x ∈ E there is a unique n element y ∈ i=1 (− 21 , 21 ] · b∗i with the property x − y ∈ L, and this element y satisfies c−1 ·
y, y ≤ min{
z, z : z ∈ E, x − z ∈ L} ≤
y, y , cn − 1 where one should read 1 n for c−1 cn −1 if c = 1. To obtain the best results in proposition 6.1, one should take c smallest possible. In section 8 we shall see that c = 43 can be achieved; that is, every lattice L has a flag F with the property li+1 (F)2 ≥ 43 · li (F)2 for 0 < i < rk L. Also, 43 is best possible in the sense that for any n > 1 there is a lattice L of rank n such that for every flag F of L there exists i with 0 < i < n and li+1 (F)2 ≤ 43 · li (F)2 . Namely, √ one can take L to be the ‘orthogonal sum’ of the hexagonal lattice Z2 + Z(1 + i 3) in C with the lattice N Zn−2 in Rn−2 , for N large enough. 7. The Reduction Graph Let L be a lattice, and let n be its rank. We write Γ(L) for the set of flags of L. We make Γ(L) into the set of vertices of a directed graph, the reduction graph of L, by drawing an arc from F = (Fi )ni=0 to F′ = (Fi′ )ni=0 if and only if there exists j, 0 < j < n, with the following properties: (i) (ii) (iii) (iv) Fi = Fi′ for all i = j; Fj + Fj′ = Fj+1 ; s(F′ ) is minimal, given (i) and (ii); s(F′ ) < s(F). Condition (iii) means, more formally, that for all flags G = (Gi )ni=0 of L satisfying Fi = Gi for all i = j and Fj +Gj = Fj+1 one has s(F′ ) ≤ s(G). To reformulate this condition, suppose that (i) and (ii) are satisfied; then we can write Fj /Fj−1 = Zw, Fj′ /Fj−1 = Zx for a certain basis w, x of the rank 2 lattice Fi+1 /Fi−1 , and (iii) is now equivalent to the inequality |
x, w| ≤
w, w/2; also, one has s(F′ )2 /s(F)2 = q(x)/q(w), so (iv) is equivalent to q(x) < q(w). We write F → F′ to denote an arc from F to F′ , and refer to it as a step in Γ(L). The length of such a step is defined to be s(F)2 /s(F′ )2 , and the number j appearing above is called the colour of the step; by (i) and (ii) it is uniquely determined. Flags and Lattice Basis Reduction 11 One readily checks that there are at most two steps in Γ(L) that start from a given flag and have a given colour; and if there are two, then they have the same length. Let K be a pure sublattice of L. The set of flags of L that comprise K may in an obvious manner be identified with Γ(K)×Γ(L/K). With this identification, one   has s (E, F) = s(E) · s(F) · d(K)(rk K) rk(L/K) , and there is a step (E, F) → (E′ , F′ ) in Γ(L) if and only if either F = F′ and there is a step E → E′ in Γ(K), or E = E′ and there is a step F → F′ in Γ(L/K); in the former case, (E, F) → (E′ , F) has the same length and colour as E → E′ , and in the latter case (E, F) → (E, F′ ) has the same length as F → F′ but the colour is larger by rk K. Proposition 7.1. Let L be a lattice. Then the map Γ(L) → Γ(L† ) sending F to F⊥ is an isomorphism of directed graphs. Corresponding steps have the same length, and their colours add up to rk L. Proof. This is entirely straightforward, and left to the reader.
The following result decribes the effect of a step on the successive lengths. Proposition 7.2. Let L be a lattice, let F → F′ be a step in Γ(L), and let j be its colour. Then one has li (F) = li (F′ ) for all i = j, j + 1, and lj+1 (F) ≤ lj (F′ ) < lj (F), lj+1 (F) < lj+1 (F′ ) ≤ lj (F) . Proof. The relation li (F) = d(Fi )/d(Fi−1 ) and (i) imply the first assertion. Write Fj /Fj−1 = Zw and Fj′ /Fj−1 = Zx, and let x̄ be the component of x orthogonal to w. Then one has lj+1 (F)2 = q(x̄) ≤ q(x) = lj (F′ )2 , and lj (F′ )2 = q(x) < q(w) = lj (F)2 . This proves the first two inequalities. The last two follow from these and the equality lj (F)lj+1 (F) = d(Fj+1 )/d(Fj−1 ) = lj (F′ )lj+1 (F′ ).
Note in particular that l1 (F′ ) ≤ l1 (F) in the situation of proposition 7.2, and, dually, ln (F′ ) ≥ ln (F). Proposition 7.3. Let L be a lattice and let F be a flag of L. Let j be an integer with 0 < j < rk L and c a real number with c ≥ 43 . Suppose that one has lj+1 (F)2 < c−1 · lj (F)2 . Then there is a step F → F′ in Γ(L) with colour j and length greater 4c than c+4 . Proof. Write Fj /Fj−1 = Zw, and choose x ∈ Fj+1 /Fj−1 with q(x) minimal subject to the condition Zx + Zw = Fj+1 /Fj−1 . With x̄ as in the previous proof, we have x = x̄ + rw with |r| ≤ 21 , so q(x) = q(x̄) + r2 q(w) = lj+1 (F)2 + r2 lj (F)2 < ′ rk L ( 1c + 41 )lj (F)2 = c+4 4c q(w) ≤ q(w). The proposition follows, with F = (Fi )i=0 ′ defined by (i) and Fj /Fj−1 = Zx.
4c will reappear in the next section. It is increasing as a The expression c+4 function of c, equal to 1 for c = 34 , and it tends to 4 for c → ∞. The parameter y 4c . A popular choice is c = 2, that appears in [3] may be viewed as the inverse of c+4 4 3 4c c+4 = 3 , y = 4 . 12 H. W. Lenstra, Jr. 8. Paths in the Reduction Graph Let L be a lattice, and put n = rk L. A path in Γ(L) is a finite sequence F1 → F2 → · · · → Ft of steps Fi → Fi+1 (1 ≤ i < t) in Γ(L); more properly, one could call this a ‘directed path’, but for ‘undirected paths’ —which would turn Γ(L) into a connected graph— we have no use. Let c be a real number with c ≥ 43 . Proposition 7.3 leads to the following procedure for transforming a given flag F of a lattice into a flag F′ satisfying the inequalities li+1 (F′ )2 ≥ c−1 · li (F′ )2 (0 < i < n) from proposition 6.1. If F itself does not satisfy these inequalities, then by proposition 7.3 one can take a step 4c from F, and iterate. Since the number of flags of size of length greater than c+4 smaller than s(F) is finite, this ‘flag reduction algorithm’ must terminate with a flag F′ with the required property. In particular, taking c = 43 , we see that we proved the statement made at the end of section 6. A good upper bound for the number of steps to be taken is of obvious interest for the analysis of actual algorithms that may be based on the procedure just described. Such a bound is easy to obtain in the case c > 43 . Namely, in that case 4c we have c+4 > 1, and since the square of the size of the flag decreases by a factor 4c in each step, the number of steps in the path F → · · · → F′ is at greater than c+4 most   log s(F)/s(F′ )  4c  2· . log c+4 This is a satisfactory bound if a good lower bound for s(F′ ) is available, which is often the case; for example, if the lattice L is such that
x, y ∈ Z for all x, y ∈ L, then one has d(K)2 ∈ Z for all sublattices K of L, so s(F′ )2 is an integer, and n s(F′ ) ≥ 1. In general, one has s(F′ ) ≥ i=1 (l/i)i/2 if l is as in proposition 3.1, by the second remark after proposition 4.3. The argument just given fails in the case c = 34 , and more generally if we allow steps of length arbitrarily close to 1. It is, for fixed rank, nevertheless possible to prove a similar logarithmic upper bound for the length of any path F → · · · → F′ in Γ(L), as we shall see in proposition 8.2. We first prove an auxiliary result on paths that consist of ‘short’ steps only. Proposition 8.1. For each integer n ≥ 0 and each real number c > 43 there is a positive integer A = A(n, c) with the following property. Let L be a lattice of rank n, and let F1 → · · · → Ft be a path in Γ(L) such that each step Fi → Fi+1 4c has length at most c+4 . Then one has t ≤ A. Proof. The proof is by induction on n, the case n ≤ 1 being trivial. Suppose that n ≥ 2, and consider a path as in proposition 8.1. We first show that there exists m ∈ {1, 2, . . . , n} with the following two properties: (i) li+1 (F1 )2 ≥ c−1 · li (F1 )2 for 0 < i < m; (ii) none of the steps Fj → Fj+1 in the path has colour m. Flags and Lattice Basis Reduction 13 If all i = 1, . . . , n − 1 satisfy the inequality in (i) then we can clearly take m = n. Now suppose that i ∈ {1, . . . , n − 1} is such that li+1 (F1 )2 < c−1 · li (F1 )2 . Then by proposition 7.3, there is a step F1 → F′ of colour i and length greater 4c 4c , so any step of colour i starting at F1 has length greater than c+4 . By than c+4 4c hypothesis, F1 → F2 has length at most c+4 , so it does not have colour i. Therefore proposition 7.2 implies li+1 (F2 ) ≤ li+1 (F1 ) and li (F2 ) ≥ li (F1 ). It follows that the inequality li+1 (F)2 < c−1 · li (F)2 , which is satisfied for F = F1 , is likewise satisfied for F = F2 ; by induction on j one now deduces that all steps Fj → Fj+1 in the path have colour different from i and that all F = Fj satisfy the inequality just stated. Hence m = i satisfies (ii). If we take for m the least value of i violating the inequality in (i), then (i) is satisfied as well. Write Fj1 for the rank 1 lattice belonging to Fj . We claim that among F11 , . . . , Ft1 there are at most (1 + 2c(n−1)/2 )n − 1 /2 different rank 1 lattices. To prove this, let m be as above. By (ii), the rank m sublattice belonging to Fj is the same for all j; let this lattice be called K. The lattices of rank at most m belonging to F1 form a flag E of K, and by (i) we have li+1 (E) ≥ c−1 · li (E)2 for 0 < i < m. Applying proposition 6.1 to K and E we see that any nonzero x ∈ K satisfies q(x)2 ≥ l = c1−m · l1 (F1 )2 . By proposition 3.1, the number of x ∈ K with m q(x) ≤ l1 (F1 )2 is at most (1 + 2c(m−1)/2 x and −x  ) . Since  generate the same lattice, it follows that K has at most (1 + 2c(m−1)/2 )m − 1 /2 sublattices M of rank 1 that satisfy d(M ) ≤ l1 (F1 ). Each Fj1 is such an M , and m ≤ n, so the claim follows. In our path, the rank 1 sublattice changes only at steps of colour 1, and at each such step the determinant of that sublattice decreases. Hence our claim implies that we can write the path F1 → · · · → Ft as the union of at most  (1 + 2c(n−1)/2 )n − 1 /2 subpaths connected by steps of colour 1, such that in each of the subpaths the rank 1 sublattice is held fixed. But when the rank 1 sublattice is held equal to M (say), one is really considering flags of the rank n − 1 lattice L/M and paths in Γ(L/M ). Application of the induction hypothesis on n now leads in a straightforward way to the   inequality in proposition 8.1, with A(n, c) = A(n − 1, c) · [ (1 + 2c(n−1)/2 )n − 1 /2].
We can now formulate and prove our main result. Proposition 8.2. For each non-negative integer n there exists a positive integer B = ′ B(n) with the following property. If L is a lattice of rank n, and  F, F are flags of L, then every path from F to F′ in Γ(L) contains at most B · 1 + log(s(F)/s(F′ )) flags. Proof. Fix a real number c with c > 43 , and call a step F1 → F2 in Γ(L) long if it 4c has length greater than 4+c , and short otherwise. Consider any path F → · · · → F′ . If k is the number of long steps, then one k    4c has s(F′ )2 ≤ 4+c ·s(F)2 , so k ·log 4+c ≤ 2 log s(F)/s(F′ ) . Hence the path is the 4c   4c union of at most 1 + 2 log(s(F)/s(F′ )) / log 4+c subpaths connected by long steps, such that each of the subpaths consists of short steps only. By proposition 8.1, the 14 H. W. Lenstra, Jr. number of flags occurring in each of the subpaths is bounded by a function of the rank. The result follows.
The proposition just proved is useful in the analysis of algorithms that involve lattices of fixed rank. When the rank varies, it becomes important to express B(n) as an explicit function of n; in particular, if one wishes such an algorithm to run in polynomial time, one may want to bound B(n) by a polynomial function of n. I do not know whether this is possible. I do know the following much weaker result. Proposition 8.3. The numbers B(n) in proposition 8.2 can be chosen such that in 3 addition one has B(n) = (4/3)n /(12+o(1)) for n → ∞. Proof. Making the proof of proposition 8.2 explicit, one finds a value for B(n) that is a function of c. One may choose c as a function of n that tends to 43 for n → ∞  o(n) 4c ; for example, one may take to be 34 sufficiently slowly for the factor log 4+c 4 1 c = 3 + n . This yields the result of proposition 8.3, but with 6 instead of 12. To achieve 12, one starts by improving proposition 8.1. In the proof of proposition 8.1, we saw that the flags in a path F1 → · · · → Ft consisting of short steps only comprise at most (1+2c(n−1)/2 )n −1 /2 different sublattices M of rank 1. One now   notes that, by duality, they also comprise at most (1+2c(n−1)/2 )n −1 /2 different sublattices N of rank n−1. It follows that there are at most (1+2c(n−1)/2 )n −3 steps of colour 1 or n − 1, and that the path is the union of at most (1 + 2c(n−1)/2 )n − 2 subpaths, connected by steps of colours 1 and n − 1, such that in each of the subpaths both N and M are fixed; it is then really a path in Γ(N/M ), where N/M has rank n − 2. In this manner, one proves that one may take A(n, c) = A(n − 2, c) · (1 + 2c(n−1)/2 )n − 2 in proposition 8.1. This improved bound leads to proposition 8.3.
Acknowledgements The author is grateful to K. I. Aardal and C. A. J. Hurkens for their comments. He was supported by the National Science Foundation under grant number DMS 9224205. References [1] K. I. Aardal, Lattice basis reduction and integer programming, rapport UU-CS1999–37, Informatica Instituut, Universiteit Utrecht, 1999. [2] H. Cohen, A course in computational algebraic number theory, Springer-Verlag, Berlin, 1993. [3] A. K. Lenstra, H. W. Lenstra, Jr. and L. Lovász, Factoring polynomials with rational coefficients, Math. Ann. 261 (1982), 515–534. [4] H. W. Lenstra, Jr., Integer programming with a fixed number of variables, report 81–03, Mathematisch Instituut, Universiteit van Amsterdam, April, 1981. Flags and Lattice Basis Reduction 15 [5] H. W. Lenstra, Jr., Integer programming with a fixed number of variables, Math. Oper. Res. 8 (1983), 538–548. Mathematisch Instituut, Universiteit Leiden, Postbus 9512, 2300 RA Leiden, The Netherlands E-mail address: hwl@math.leidenuniv.nl Department of Mathematics # 3840, University of California, Berkeley, CA 94720–3840, U. S. A.