Handbook of Algorithms and Data Structures Terre eh ce) OMG e@ rnd R. Baeza-YatesHandbook of Algorithms and Data Structures In Pascal and C Second EditionINTERNATIONAL COMPUTER SCIENCE SERIES Consulting editors AD McGettrick University of Strathclyde Jvan Leeuwen University of Utrecht SELECTED TITLES IN THE SERIES Programming Language Translation: A Practical Approach PD Terry Data Abstraction in Programming Languages J M Bishop The Specification of Computer Programs = W M Turski and T S E Maibaum Syntax Analysis and Software Tools KJ Gough Functional Programming A J Field and P G Harrison The Theory of Computability: Programs, Machines, Effectiveness and Feasibility R Sommerhalder and S C van Westrhenen An Introduction to Functional Programming through Lambda Calculus G Michaelson High-Level Languages and their Compilers =D Watson Programming in Ada (3rd Edn) J G P Barnes Elements of Functional Programming C Reade Software Development with Modula-2. _D Budgen . Program Derivation: The Development of Programs from Specifications | R G Dromey Object-Oriented Programming with Simula_—B Kirkerud Program Design with Modula-2 _S. Eisenbach and C Sadler Real Time Systems and Their Programming Languages A Burns and A Wellings Fortran 77 Programming (2ndEdn) TMR Ellis Prolog Programming for Artificial Intelligence (2nd Edn) ‘I Bratko Logic for Computer Science S$ Reeves and M Clarke Computer Architecture M De Blasi The Programming Process J T Latham, V J Bush and I D Cottam’Handbook of Algorithms and ara Structur ata JLIULLU In Pascal and C Second Edition TY LV g G.H. Gonnet ETH, Zurich ReBaczanLvates fifdrsicy of GHile? Santiago A vv ADDISON -WESLEY PUBLISHING COMPANY Wokingham, England Reading, Massachusetts e Menlo Park, California e New York Don Mills, Ontario Amsterdam e Bonn e Sydney « Singapore Tokyo* Madrid * San Juan» Milane Paris « Mexico City » Scoul * Taipei© 1991 Addison-Wesley Publishers Ltd. © 1991 Addison-Wesley Publishing Company Inc. All rights reserved. No part of this publication may be reproduced, stored ina retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without prior written permission of the publisher. The programs in this book have been included for their instructional value. They have been tested with care but are not guarantecd for any particular purpose. The publisher does not offer any warranties or representations, nor does it accept any liabilities with respect to the programs. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Addison-Wesley has made every attempt to supply trademark information about manufacturers and their products mentioned in this book. A list of the trademark designations and their owners appears on p. xiv. Cover designed by Crayon Design of Henley-on-Thames and printed by The Riverside Printing Co. (Reading) Ltd. Printed in Great Britain by Mackays of Chatham pic, Chatham, Kent First edition published 1984. Reprinted 1985. Second edition printed 1991. Reprinted 1991. British Library Cataloguing in Publication Data Gonnet, G. H. (Gaston H.) Handbook of algorithms and data structures : in Pascal and C.-2nd. ed. 1. Programming. Algorithms I. Title II. Baeza-Yates, R. (Ricardo) © 005.1 ISBN 0-201-41607-7 Library of Congress Cataloging in Publication Data Gonnet, G. H. (Gaston H.) Handbook of algorithms and data structures : in Pascal and C/ G.H. Gonnet, R. Baeza- Yates. - - 2nd ed. p. cm. - - (International computer science series) Includes bibliographical references (p. _) and index. ISBN 0-201-41607-7 1, Pascal (Computer program language) 2. (Computer program language) 3. Algorithms. 4. Data structures (Computer science) I, Baeza-Yates, R. (Ricardo) II. Title. III. Series. QA76.73.P2G66 1991 90-26318 005. 13’3--de20 CIPmy girls: Ariana and MartaPreface Preface to the first edition Computer Science has been, throughout its evolution, more an art than a sci- ence. My favourite example which illustrates this point is to compare a major software project (like the writing of a compiler) with any other major project (like the construction of the CN tower in Toronto). It would be absolutely unthinkable to let the tower fall down a few times while its design was being debugged: even worse would be to open it to the public before discovering some other fatal flaw. Yet this mode of operation is being used everyday by almost everybody in software production. Presently it is very difficult to ‘stand on your predecessor’s shoulders’, most of the time we stand on our predecessor’s toes, at best. This handbook was written with the intention of making available to the computer scien- tist, instructor or programmer the wealth of information which the field has generated in the last 20 years. Most of the results are extracted from the given references. In some cases the author has completed or generalized some of these results. Accuracy is certainly one of our goals, and consequently the author will cheerfully pay $2.00 for each first report of any type of error appearing in this handbook. Many people helped me directly or indirectly to complete this project. Firstly I owe my family hundreds of hours of attention. All my students and colleagues had some impact. In particular I would like to thank Maria Carolina Monard, Nivio Ziviani, J. lan Munro, Per-Ake Larson, Doron Rotem and Derick Wood. Very special thanks go to Frank W. Tompa who is also the coauthor of chapter 2. The source material for this chapter appears in a joint paper in the November 1983 issue of Communications of the ACM. Montevideo G.H. Gonnet December 1983 viiviii PREFACE Preface to the second edition The first edition of this handbook has been very well received by the com- munity, and this has given us the necessary momentum for writing a second edition. In doing so, R. A. Baeza-Yates has joined me as a coauthor. Without his help this version would have never appeared. This second edition incorporates many new results and a new chapter on text searching. The area of text managing, in particular searching, has risen in importance and matured in recent times. The entire subject of the handbook has matured too; our citations section has more than doubled in size. Table searching algorithms account for a significant part of this growth. Finally we would like to thank the over one hundred readers who notified us about errors and misprints, they have helped us tremendously in correcting all sorts of blemishes. We are especially grateful for the meticulous, even amazing, work of Lynne Balfe, the proofreader. We will continue cheerfully to pay $4.00 (increased due to inflation) for each first report of an error. Zurich G.H. Gonnet December 1990 Santiago de Chile R.A. Baeza-Yates December 1990Contents Preface 1 Introduction Structure of the chapters Naming of variables Probabilities Asymptotic notation About the programming languages On the code for the algorithms Complexity measures and real timings 2 Basic Concepts 2.1 2.2 Data structure description 2.1.1 Grammar for data objects 2.1.2 Constraints for data objects 2.1.2.1 Sequential order 2.1.2.2 Uniqueness 2.1.2.3 Hierarchical order 2.1.2.4 Tlierarchical balance 2.1.2.5 Optimality Algorithm descriptions 2.2.1 Basic (or atomic) operations 2.2.2 Building procedures 2.2.2.1 Composition 2.2.2.2 Alternation 2.2.2.3 Conformation 2.2.2.4 Self-organization 2.2.3 Interchangeability < B: NO PRworex CONTENTS 3 Searching Algorithms 3.1 3.2 3.3 3.4 Sequential search 3.1.1 Basic sequential search 3.1.2 Self-organizing sequential search: move-to-front method 3.1.3 Self-organizing sequential search: transpose method 3.1.4 Optimal sequential search 3.1.5 Jump search Sorted array search 3.2.1 Binary search 3.2.2 Interpolation search 3.2.3. Interpolation—-sequential search Hashing 3.3.1 Practical hashing functions 3.3.2 Uniform probing hashing 3.3.3 Random probing hashing 3.3.4 Linear probing hashing 3.3.5 Double hashing 3.3.6 Quadratic hashing 3.3.7 Ordered and split-sequence hashing 3.3.8 Reorganization schemes 3.3.8.1 Brent’s algorithm 3.3.8.2 Binary tree hashing 3.3.8.3 Last-come-first-served hashing 3.3.8.4 Robin Hood hashing 3.3.8.5 Self-adjusting hashing 3.3.9 Optimal hashing 3.3.10 Direct chaining hashing 3.3.11 Separate chaining hashing 3.3.12 Coalesced hashing 3.3.13 Extendible hashing 3.3.14 Linear hashing 3.3.15 External hashing using minimal internal storage 3.3.16 Perfect hashing 3.3.17 Summary Recursive structures search 3.4.1 Binary tree search 3.4.1.1 Randomly generated binary trees 3.4.1.2 Random binary trees 3.4.1.3 Height-balanced trees 3.4.1.4 Weight-balanced trees 3.4.1.5 Balancing by internal path reduction 3.4.1.6 Heuristic organization schemes on binary trees 3.4.1.7 Optimal binary tree search 3.4.1.8 Rotations in binary trees 3.4.1.9 Deletions in binary trees3.5 3.4.1.10 m-ary search trees 3.4.2 B-trees 3.4.2.1 2-3 trees 3.4.2.2 Symmetric binary B-trees 3.4.2.3 1-2 trees 3.4.2.4 2-3-4 trees 3.4.2.5 B-tree variations 3.4.3 Index and indexed sequential files 3.4.3.1 Index sequential access method 3.4.4 Digital trees 3.4.4.1 Hybrid tries 3.4.4.2 Tries for word-dictionaries 3.4.4.3 Digital search trees 3.4.4.4 Compressed tries 3.4.4.5 Patricia trees Multidimensional scarch 3.5.1 Quad trees 3.5.1.1 Quad tries 3.5.2 K-dimensional trees 4 Sorting Algorithms 4.1 4.2 4.3 Techniques for sorting arrays 4.1.1 Bubble sort 4.1.2 Linear insertion sort 4.1.3 Quicksort 4.1.4 Shellsort 4.1.5 Heapsort 4.1.6 Interpolation sort 4.1.7 Linear probing sort 4.1.8 Summary Sorting other data structures 4.2.1 Merge sort 4.2.2 Quicksort for lists 4.2.3 Bucket sort 4.2.4 Radix sort 4.2.5 Hybrid methods of sorting 4.2.5.1 Recursion termination 4.2.5.2 Distributive partitioning 4.2.5.3 Non-recursive bucket sort 4.2.6 Treesort Merging 4.3.1 List merging 4.3.2 Array merging 4.3.3 Minimal-comparison merging CONTENTS xi 116 117 124 126 128 129 130 120 40u 132 133 137 138 138 140 140 143 144 146 149 153 153 154 156 158 161 164 166 168 170 171 173 174 176 179 180 181 181 182 182 183 184 185 186xii CONTENTS 44 External sorting 4.4.1 Selection phase techniques 44.1.1 Replacement selection 4.4.1.2 Natural selection 44.1.3 Alternating selection 4.4.1.4 Merging phase 4.4.2 Balanced merge sort 44.3 Cascade merge sort 4.4.4 Polyphase merge sort 4.4.5 Oscillating merge sort 44.6 External Quicksort 5 Selection Algorithms 5.1 5.2 Priority queues 5.1.1 Sorted/unsorted lists 5.1.2 P-trees 5.1.3 Heaps 5.1.4 Van Emde-Boas priority queues 5.1.5 Pagodas 5.1.6 Binary trees used as priority queues 5.1.6.1 Leftist trees 5.1.6.2 Binary priority queues 5.1.6.3 Binary search trees as priority queues 5.1.7 Binomial queues 5.1.8 Summary Selection of kth element 5.2.1 Selection by sorting 5.2.2 Selection by tail recursion 5.2.38 Selection of the mode 6 Arithmetic Algorithms 6.1 6.2 6.3 6.4 Basic operations, multiplication/division Other arithmetic functions 6.2.1 Binary powering 6.2.2 Arithmetic-geometric mean 6.2.3 Transcendental functions Matrix multiplication 6.3.1 Strassen’s matrix multiplication 6.3.2 Further asymptotic improvements Polynomial evaluation 187 189 189 190 191 192 193 195 196 200 201 205 205 206 209 211 216 218 221 991 22h 223 225 226 227 228 230 230 232 235 240 240 242 243 245 246 247 248CONTENTS xiii 7 Text Algorithms 251 7.1 Text searching without preprocessing 251 7.1.1 Brute force text searching 253 7.1.2 Knuth-Morris-Pratt text searching 254 7.1.3 Boyer-Moore text searching 256 7.1.4- Searching sets of strings : 259 7.1.5 Karp-Rabin text searching 260 7.1.6 Searching text with automata 262 7.1.7 Shift-or text searching 266 7.1.8 String similarity searching 267 7.1.9 Summary of direct text searching 270 7.2 Searching preprocessed text 270 7.2.1 Inverted files 271 7.2.2 Trees used for text searching 273 7.2.3° Searching text with automata 275 7.2.4 Suffix arrays and PAT arrays 277 725 DAWG 279 7.2.6 Hashing methods for text searching 280 7.2.7 P-strings 281 7.3 Other text searching problems 283 7.3.1 Searching longest common subsequences 283 7.3.2 Two-dimensional searching 284 I Distributions Derived from Empirical Observation 289 1.1. Zipf’s law 289 1.1.1 First generalization of a Zipfian distribution 290 1.1.2 Second generalization of a Zipfian distribution 290 1.2 Bradford’s law 291 1.3 Lotka’s law 293 1.4 80%-20% rule 293 II Asymptotic Expansions 297 II.1 Asymptotic expansions of sums 298 II.2 Gamma-type expansions 300 11.3 Exponential-type expansions 301 II.4 Asymptotic expansions of sums and definite integrals contain- ing e~™” 302 II.5 Doubly exponential forms 303 IL.6 Roots of polynomials 304 II.7 Sums containing descending factorials 305 II.8 Summation formulas 307 III References 309 III.1 Textbooks 309 III.2 Papers 311xiv CONTENTS IV Algorithms coded in Pascal and C 375 IV.1 Searching algorithms 375 IV.2 Sorting algorithms 387 IV.3 Selection algorithms 399 IV.4 Text algorithms 408 Index 415 Trademark notice SUN 3™ and SunOS™ are trademarks of Sun Microsystems, Inc.| Introduction This handbook is intended to contain most of the information available on algorithms and their data structures; thus it is designed to serve a wide spec- trum of users, from the programmer who wants to code efficiently to the student or researcher who needs information quickly. The main emphasis is placed on algorithms. For these we present their description, code in one or more languages, theoretical results and extensive lists of references. : 1.1 Structure of the chapters The handbook is organized by topics. Chapter 2 offers a formalization of the description of algorithms and data structures; Chapters 3 to 7 discuss search- ing, sorting, selection, arithmetic and text algorithms respectively. Appendix I describes some probability distributions encountered in data processing; Ap- pendix II contains a collection of asymptotic formulas related to the analysis of algorithms; Appendix III contains the main list of references and Appendix IV contains alternate code for some algorithms. The chapters describing algorithms are divided into sections and subsec- tions as needed. Each algorithm is described in its own subsection, and all have roughly the same format, though we may make slight deviations or omis- sions when information is unavailable or trivial. The general format includes: (1) Definition and explanation of the algorithm and its classification (if ap- plicable) according to the basic operations described in Chapter 2. (2) Theoretical results on the algorithm’s complexity. We are mainly inter- ested in measurements which indicate an algorithm’s running time and 12 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES its space requirements. Useful quantities to measure for this information include the number of comparisons, data accesses, assignments, or ex- changes an algorithm might make. When looking at space requirements, we might consider the number of words, records, or pointers involved in an implementation. Time complexity covers a much broader range of measurements. For example, in our examination of searching algo- rithms, we might be able to attach meaningful interpretations to most of the combinations of the query average comparisons addarecordinto variance accesses deletearecord from minimum number of assignments when we modify arecordof worstcase exchanges reorganize average w.c. function calls build read sequentially (3) (4) (5) (6) the structure. Other theoretical results may also be presented, such as enumerations, generating functions, or behaviour of the algorithm when the data elements are distributed according to special distributions. The algorithm. We have selected Pascal and C to describe the algo- rithms. Algorithms that may be used in practice are described in one or both of these languages. For algorithms which are only of theoretical interest, we do not provide their code. Algorithms which are coded both in Pascal and in C will have one code in the main text and the other in Appendix IV. Recommendations. Following the algorithm description we give several hints and tips on how to use it. We point out pitfalls to avoid in coding, suggest when to use the algorithm and when not to, say when to expect best and worst performances, and provide a variety of other comments. Tables. Whenever possible, we present tables which show exact values of complexity measures in selected cases. These are intended to give a feeling for how the algorithm behaves. | When precise theoretical results are not available we give simulation results, generally in the form zzz + yy where the value yy is chosen so that the resulting interval has a confidence level of 95%. In other words, the actual value of the complexity measure falls out of the given interval only once every 20 simulations. Differences between internal and external storage. Some algorithms may perform better for internal storage than external, or vice versa. When this is true, we will give recommendations for applications in each case. Since most of our analysis up to this point will implicitly assume that internal memory is used, in this section we will look more closely at the external case (if appropriate). We analyze the algorithm’s behaviourINTRODUCTION 3 when working with external storage, and discuss any significant practical considerations in using the algorithm externally. (7) With the description of each algorithm we include a list of relevant references. General references, surveys, or tutorials are collected at the end of chapters or sections. The third appendix contains an alphabetical list of all references with cross-references to the relevant algorithms. 1.2 Naming of variables The naming of variables throughout this handbook is a compromise between uniformity of notation and accepted terminology in the specific areas. Except for very few exceptions, explicitly noted, we use: n for the number of objects or elements or components in a structure; m for the size of a structure; 6 for bucket sizes, or maximum number of elements in a physical block; d for the digital cardinality or size of the alphabet. The complexity measures are also named uniformly throughout the hand- book. Complexity measures are named XZ and should be read as ‘the number of Xs performed or needed while doing Z onto a structure of size n’. Typical values for X are: : accesses, probes or node inspections; : comparisons or node inspections; : external accesses; : height of a recursive structure (typically a tree); : iterations (or number of function calls); : length (of path or longest probe sequence); M : moves or assignments (usually related to record or key movements); T : running time; S': space (bytes or words). BN by Q Typical values for Z are: null (no superscript): suce is only one possibility); ’ unsuccessful search; C : construction (building) of structure; D: deletion of an element; E : extraction of an element (mostly for priority queues); I: insertion of a new element; ° 8 é a i i bad4 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES M : merging of structures; Opt : optimal construction or optimal structure (the operation is usually implicit); MM : minimax, or minimum number of X’s in the worst case: this is usually used to give upper and lower bounds on the complexity of a problem. Note that X/ means number of operations done to insert an element into a structure of size n or to insert the n + 1st element. Although these measures are random variables (as these depend on the particular structure on which they are measured), we will make exceptions for Cn and C’, which most of the literature considers to be expected values. 1.3 Probabilities The probability of a given event is denoted by Pr{event}. Random vari- ables follow the convention described in the preceding section. The expected value of a random variable X is written E[X] and its variance is o?(X). In particular, for discrete variables X E[X] = 4, = LiPr{xX =i} o°(X) = SOP Pr{X =i} - EXP = B[X?] - E[XP We will always make explicit the probability universe on which expected values are computed. This is ambiguous in some cases, and is a ubiquitous problem with expected values. : To illustrate the problem without trying to confuse the reader, suppose that we fill a hashing table with keys and then we want to know about the average number of accesses to retrieve one of the keys. We have two potential probability universes: the key selected for retrieval (the one inserted first, the one inserted second, ...) and the actual values of the keys, or their probing sequence. We can compute expected values with respect to the first, the second, or both universes. In simpler terms, we can find the expected value of any key for a given file, or the expected value of a given key for any file, or the expected value of any key for any file. Unless otherwise stated, (1) the distribution of our elements is always random independent uniform U(0,1); (2) the selection of a given element is uniform discrete between all possible elements; (3) expected values which relate to multiple universes are computed with respect to all universes. In terms of the above example, we will compute expected values with respect to randomly selected variables drawn from a uniform U(0, 1) distribution.INTRODU TION 5 1.4 Asymptotic notation Most of the complexity measures in this handbook are asymptotic in the size of the problem. The asymptotic notation we will use is fairly standard and ix given below: F(n) = O(g(n)) implies that there exists and no such that | f(n) |< kg(n) for n > ny. n) = o(g(n)) > li f(r) _ fl) = ofg(n)) — tim A = 0 F(n) = O(g(n)) implies that there exists ky, kz, (k1 xk2 > 0) and no such that kig(n) < flny< kog(n) for n > no, or equivalently that f(n) = O(g(n)) and g(r) = O(f(n)). f(n) = Ag(n)) + g(n) = O(f(n)) f(n) = w(g(n)) > g(n) = o(f(n)) f(n) ® g(n) — f(n)—g(n) = o(g(n)) We will freely use arithmetic operations with the order notation, for ex- ample, z F(n) = h(n) + O(9(n)) means F(n) — h(n) = O(g(n)) Whenever we write f(n) = O(g(n)) it is with the understanding that we know of no better asymptotic bound, that is, we know of no h(n) = o(g(n)) such that f(n) = O(h(n)). 1.5 About the programming languages We use two languages to code our algorithms: Pascal and C. After writing many algorithms we still find situations for which neither of these languages present a very ‘clean’ or understandable code. Therefore, whenever possible, we use the language which presents the shortest and most readable code. We intentionally allow our Pascal and C style of coding to resemble each other. | A minimal number of Pascal programs contain goto statements. These statements are used in place of the equivalent C statements return and reak, and are correspondingly so commented. Indeed we view their absence6 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES from Pascal as a shortcoming of the language. Another irritant in coding some algorithms in Pascal is the lack of order in the evaluation of logical ex- pressions. This is unfortunate since such a feature makes algorithms easier to understand. The typical stumbling block is while (p <> nil) and (key <> pf.k) do... Such a statement works in C if we use the sequential and operator (dd), but for Pascal we have to use instead: while p <> nil do begin if key = p}.k then goto 999 {*#« break xx } ; 999: Other minor objections are: the inability to compute addresses of non- heap objects in Pascal (which makes treatment of lists more difficult); the lack of variable length strings in Pascal; the lack of a with statement in C; and the lack of var parameters in C. (Although this is technically possible to overcome, it obscures the algorithms.) Our Pascal code conforms, as fully as possible, to the language described in the Pascal User Manual and Report by K. Jensen and N. Wirth. The C code conforms to the language described in The C Programming Language by B.W. Kernighan and D.M. Ritchie. 1.6 On the code for the algorithms Except for very few algorithms which are obviously written in pseudo-code, the algorithms in this handbook were run and tested under two different compilers. Actually the same text which is printed is used for compiling, for testing, for running simulations and for obtaining timings. This was done in an attempt to eliminate (or at least drastically reduce!) errors. Each family of algorithms has a ‘tester set’? which not only checks for correct behaviour of the algorithm, but also checks proper handling of limiting conditions (will a sorting routine sort a null file? one with one element? one with all equal keys? ...). In most cases the algorithms are described as a function or a procedure or a small set of functions or procedures. In a few cases, for very simple algorithms, the code is described as in-line code, which could be encapsulated in a procedure or could be inserted into some other piece of code. Some algorithms, most notably the searching algorithms, are building blocks or components of other algorithms or programs. Some standard actions should not be specified for the algorithm itself, but rather will be specified once that the algorithm is ‘composed’ with other parts (chapter 2 definesINTRODUCTION 7 composition in more detail). A typical example of a standard action is an error condition. The algorithms coded for this handbook always use the same names for these standard actions. Error detection of an unexpected condition during execution. Whenever Error is encountered it can be substituted by any block of statements. For example our testers print an appropriate message. found(record) function call that is executed upon completion of a successful search. Its argument is a record or a pointer to a record which contains the searched key. notfound(key) function called upon an unsuccessful search. Its argument is the key which was not found. A special effort has been made to avoid duplication of these standard actions for identical conditions. This makes it easier to substitute blocks of code for them. 1.7 Complexity measures and real timings For some families of algorithms we include a comparison of real timings. These timings are to be interpreted with caution as they reflect only one sample point in the many dimensions of hardwares, compilers, operating systems, and so on. Yet we have equally powerful reasons to present at least one set of real camplexities com) plexities. The main reasons for including real timing comparisons are that they take into account: (1) the actual cost of operations, (2) hidden costs, such as storage allocation, and indexing. The main objections, or the factors which may invalidate these real timing tables, are: (1) the results are compiler dependent: although the same compiler is used for each language, a compiler may favour one construct over others; (2) the results are hardware dependent; (3) in some cases, when large amounts of memory are used, the timings may be load dependent. The timings were done on a Sun 3 running the SunOS 4.1 operating system. Both C and Pascal compilers were run with the optimizer, or object code improver, to obtain the best implementation for the algorithms. There were no attempts made to compare timings across languages. Ali the timing results are computed relative to the fastest algorithm. To avoid the incidence of start up-costs, loading, and so on, the tests were run on problems8 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES of significant size. Under these circumstances, some O(n?) algorithms appear to perform very poorly.D Basic Concepts 2.1 Data structure description The formal description of data structure implementations is similar to the formal description of programming languages. In defining a programming language, one typically begins by presenting a syntax for valid programs in the form of a grammar and then sets further validity restrictions (for example, usage rules for symbolic names) which give constraints that are not captured by the grammar. Similarly, a valid data structure implementation will be one that satisfies a syntactic grammar and also obeys certain constraints. For example, for a particular data structure to be a valid weight-balanced binary tree, it must satisfy the grammatical rules for binary trees and it must also satisfy a specific balancing constraint. 2.1.1 Grammar for data objects A sequence of real numbers can be defined by the BNF production s:= [real , ] | nil Thus a sequence of reals can have the form nil, [real,nil], [real,[real,nil]], and so on. Similarly, sequences of integers, characters, strings, boolean constants, could be defined. However, this would result in a bulky collection of production rules which are all very much alike. One might first try to eliminate this repetitiveness by defining s= [, ] | nil where is given as the list of data types ::= real | int | bool | string | char 9However, this pair of productions generates unwanted sequences such as (real, [int,nil]] as well as the homogeneous sequences desired. To overcome this problem, the syntax of a data object class can be defined using a W-grammar (also called a two-level or van Wijngaarden grammar). Actually the full capabilities of W-grammars will not be utilized; rather the syntax will be defined using the equivalent of standard BNF productions together with the uniform replacement rule as described below. A W-grammar generates a language in two steps (levels). In the first step, a collection of generalized rules is used to create more specific production tules. In the second step, the production rules generated in the first step are used to define the actual data structures. First, the problem of listing repetitive production rules is solved by starting out with generalized rule-forms known as hyperrules, rather than the rules themselves. The ceneralized form of a sequence & is siven by the bvnerriule WNEMSSIVES. LS FENCTAuZEG 10TM G1 @ Sequence © iS Ziven oy the nyperrwe s—D: [D,s—D]; nil The set of possible substitutions for D are now defined in a meta- production, as distinguished from a conventional BNF-type production. For example, if D is given as D :: real; int; bool; string; char; --- a sequence of real numbers is defined in two steps as follows. The first step consists of choosing a value to substitute for D from the list of possibilities given by the appropriate metaproduction; in this instance, D — real. Next invoke the uniform replacement rule to substitute the string real for D ev- erywhere it appears in the hyperrule that defines s— D. This substitution gives s—real : [real, s — real]; nil Thus the joint use of the metaproduction and the hyperrule generates an ordi- nary BNF-like production defining real sequences. The same two statements can generate a production rule for sequences of any other valid data type (integer, character, ...). Figures 2.1 and 2.2 contain a W-grammar which will generate many con- ventional data objects. As further examples of the use of this grammar, consider the generation of a binary tree of real numbers. With D — real and LEAF — nil, HR[3] generates the production rule bt — real — nil : [real , bt — real — nil, bt — real — nil ] ; nil Since bt — real — nil is one of the legitimate values for D according to M[1] let D — bt — real — nil from which HR[1] indicates that such a binary tree is a legitimate data structure.Metaproductions M(t] D: M[2] DICT :: M{3] REC M[4] LEAF M[5] N:: M[6é] DIGIT M(7] KEY Cecon dly yeconay ci chaining. The production real; int; bool; string; char;...; (Dy; REC; (REC); [D]; 3—D; gt -D-LEAF; DICT; {KEY}N, s-— KEY; bt — KEY — LEAF; mt — N—KEY — LEAF; tr -N-KEY. D; D, REC. nil; D. DIGIT; DIGIT N. 0; 1; 2; 3; 4; 5; 6; 7; 8; 9. real; int; string; char; ( iY, REC). BASIC CONCEPTS 11 # atomic data types # array # record # reference # sequence # general tree : # dictionary structures # other.structure classes such as graphs, sets, Priority queues. # sequential search # binary tree # multiway tree # digital tree # record definition # search key Figure 2.1: Metaproductions for data objects. onsider oOnsiaer s — (string, int) : [(string, int), s — (string,int)]; nil and M[1] yield D = {s— (string, int)}8° Thus HR[1] will yield a production for an array of sequences of string/integer pairs usable, for example, to record NAME/AGE entries using hashing. Finally consider a production rule for structures to contain B-trees (Section 3.4.2) of strings using HR[4] and the appropriate metaproductions to yield mt — 10 — string — nil : [int, {string}19, {mt — 10 — string — nil} 4% ; nil12 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES Hyperrules HR{1] datastructure: D. HR[2] s-D: [D,s—D]; nil. HR{3] bt-D-LEAF: [D, bt-D-LEAF, bt -D-LEAF]; LEAF. HR[4] mt-N-D-LEAF: [int, (p}N ,»{mt-N-D- LEAF}4]; LEAF. HR{s] gt-D-LEAF: [De gt D LEAF]; LEAF. HR{6] tr-N-D: [{tr-N-D}¥ J; [D]; nil. Figure 2.2: Hyperrules for data objects. In this multitree, each node contains 10 keys and has 11 descendants. Certain restrictions on B-trees, however, are not included in this description (that the number of actual keys is to be stored in the int field in each node, that this number must be between 5 and 10, that the actual keys will be stored contiguously in the keys-array starting at position 1, ...); these will instead be defined as constraints (see below). The grammar rules that we are using are inherently ambiguous. This is not inconvenient; as a matter of fact it is even desirable. For example, consider D — {DIN = {reai}}° (2.1) and D — DICT — {KEY}N — {real}1® (2.2) Although both derivation trees produce the same object, the second one de- scribes an array used as a sequential implementation of a dictionary structure, while the first may just be a collection of real numbers. In other words, the derivation tree used to produce the data objects contains important semantic information and should not be ignored. 2.1.2. Constraints for data objects Certain syntactic characteristics of data objects are difficult or cumbersome to defn. mantic le define using formal grammars. A semantic rule or constraint may be regarded as a boolean function on data objects (S : D — bool) that indicates which are valid and which are not. Objects that are valid instances of a data structure implementation are those in the intersection of the set produced by the W- grammars and those that satisfy the constraints. Below are some examples of semantic rules which may be imposed on data structures. As phrased, these constraints are placed on data structures that have been legitimately produced by rules given in the previous section.BASIC CONCEPTS. 13 2.1.2.1 Sequential order Many data structures are kept in some fixed order (for example, the records in a file are often arranged alphabetically or numerically according to some key). Whatever work is done on such a file should not disrupt this order. This definition normally applies to s— D and {D}X. 2.1.2.2 Uniqueness Often it is convenient to disallow duplicate values in a structure, for example in representing sets. At other times the property of uniqueness can be used to ensure that records are not referenced several times in a structure (for example, that a linear chain has no cycles or that every node in a tree has only one parent). 2.1.2.3 Hierarchical order For all nodes, the value stored at any adjacent node is related to the value at the node according to the type of adjacency. This definition normally applies to bt - D — LEAF, mt - N—D —- LEAF and gt - D- LEAF. Lexicographical trees A lexicographical tree is a tree that satisfies the following condition for every node s: if s has n keys (key1, keys, ..., keyn) stored in it, s must have n + 1 descendant subtrees to,t1,...,tn. Furthermore, if dp is any key in any node of to, d; any key in any node of t;, and so on, the inequality dy < key, < dy <...< keyn < dy must hold. Priority queues A priority queue can be any kind of recursive structure in which an order relation has been established between each node and its descendants. One example of such an order relation would be to require that keyp < keyg, where keyp is any key in a parent node, and keyg is any key in any descendant of that node. 2.1.2.4 Hierarchical balance Height balance Let s be any node of a tree (binary or multiway). Define h(s) as the height of the subtree rooted in s, that is, the number of nodes in the tallest branch starting at s. One structural quality that may be required is that the height of a tree along any pair of adjacent branches be approximately the same. More formally, the height balance constraint is | h(si) — h(s2) | < 6 where s; and S2 are any two subtrees of any node in the tree, and 6 is a constant giving14 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES the maximum allowable height difference. In B-trees (see Section 3.4.2) for example, 6 = 0, while in AVL-trees § = 1 (see Section 3.4.1.3). Weight balance For any tree, the weight function w(s) is defined as the number of external nodes (leaves) in the subtree rooted at s. A weight balance condition requires that for any two nodes s; and sg, if they are both subtrees of any other node in the tree, r < w(s1)/w(s2) < 1/r where r is a positive constant less than 1. 2.1.2.5 Optimality Any condition on a data structure which minimizes a complexity measure (such as the expected number of accesses or the maximum number of com- parisons) is an optimality condition. If this minimized measure of complexity is based on a worst-case value, the value is called the minimax; when the minimized complexity measure is based on an average value, it is the minave. In summary, the W-grammars are used to define the general shape or pattern of the data objects. Once an object is generated, its validity is checked against the semantic rules or constraints that may apply to it. References: [Pooch, U.W. et al., 73], [Aho, A.V. et al., 74], [Rosenberg, A.L., 74], [Rosen- berg, A.L., 75], [Wirth, N., 76], [Claybrook, B.G., 77], [Hollander, C.R., 77], [Honig, W.L. et al., 77], [MacVeigh, D.T., 77], [Rosenberg, A.L. et al., 77], [Cremers, A.B. et al., 78], [Gotlieb, C.C. et al., 78], [Rosenberg, A.L., 78], [Bo- brow, D.G. et al., 79], [Burton, F.W., 79], [Rosenberg, A.L. et al., 79], [Rosen- berg, A.L. et al., 80], [Vuillemin, J., 80], [Rosenberg, A.L., 81], [O’Dunlaing, C. et al., 82], [Gonnet, G.H. et al., 83], [Wirth, N., 86]. 2.2 Algorithm descriptions Having defined the objects used to structure data, it is appropriate to de- scribe the algorithms that access them. Furthermore, because data objects are not static, it is equally important to describe data structure manipulation algorithms. An algorithm computes a function that operates on data structures. More formally, an algorithm describes a map S + R or S x P > R, where S, P, and R are all data structures; S is called the input structure, P contains parameters (for example, to specify a query), and R is the result. The two following examples illustrate these concepts: (1) Quicksort is an algorithm that takes an array and sorts it. Since there are no parameters, .BASIC CONCEPTS 15 Quicksort: array — sorted-array (2) B-tree insertion is an algorithm that inserts a new record P into a B-tree S, giving a new B-tree as a result. In functional notation, B-tree-insertion: B-tree x new-record — B-tree Algorithms compute functions over data structures. As always, different algorithms may compute the same functions; sin(2x) and 2sin(zx) cos(x) are two expressions that compute the same function. Since equivalent algorithms have different computational requirements however, it is not merely the func- tion computed by the algorithm that is of interest, but also the algorithm itself. In the following section, we describe a few basic operations informally in order to convey their flavour. References: [Aho, A.V. et al., 74], [Wirth, N., 76], [Bentley, J.L., 79], [Bentley, J.L., 79], [Saxe, J.B. et al., 79], [Bentley, J.L. et al., 80], [Bentley, J.L. et al., 80], [Remy, J.L., 80], [Mehlhorn, K. et al., 81], (Overmars, M.H. et al., 81], [Overmars, M.H. et al., 81], [Overmars, M.H. et al., 81], [Overmars, M.H. et al., 81], [Overmars, M.H., 81], [Rosenberg, A.L., 81], [Overmars, M.H. et al., 82], [Gonnet, G.H. et al., 83], [Chazelle, B. et al., 86], [Wirth, N., 86], [Tarjan, R.E., 87], [Jacobs, D. et al., 88], [Manber, U., 88], [Rao, V.N.S. et al., 88], (Lan, K.K., 89], [Mehlhorn, K. et al., 90]. 2.2.1. Basic (or atomic) operations A primary class of basic operations manipulate atomic values and are used to focus an algorithm’s execution on the appropriate part(s) of a composite data object. The most common of these are as follows: Selector and constructor A selector is an operation that allows access to any of the elements corre- sponding to the right-hand side of a production rule from the corresponding left-hand side object. A constructor is an operation that allows us to assemble an element on the left-hand side of a production given all the corresponding elements on the right. For example, given a {string}9 and an integer, we can select the ith element, and given two bt — real — nil and a real we can construct a new bt — real — nil. Replacement non-scalar x selector x value > non-scalar A replacement operator removes us from pure functions by introducing the assignment statements. This operator introduces the possibility of cyclic and shared structures. For example, given a bt-D-LEAF we can forma threaded16 HANDBOOK OF ALGORITHMS AND DATA STRUCTURES binary tree by replacing the nil values in the leaves by (tagged) references back to appropriate nodes in the tree. Ranking set of scalars x scalar > integer This operation is defined on a set of scalars X1,X2,...,Xn and uses another scalar X as a parameter. Ranking determines how many of the Xj; values are less than or equal to X, thus determining what rank X would have if it were ordered with the other values. More precisely, ranking is finding an integer i such that there is a subset A C {X1,Xo,...,X,} for which | A] = 7 and X; € A if and only if X; < X. Ranking is used primarily in directing multiway decisions. For example, in a binary decision, n = 1, and ¢ is zero if X < Xj, one otherwise. Hashing value x range — integer Hashing is an operation which normally makes use of a record key. Rather than using the actual key value however, an algorithm invokes hashing to transform the key into an integer in a prescribed range by means of a hashing function and then uses the generated integer value. Interpolation numeric-value x parameters > integer Similarly to hashing, this operation is typically used on record keys. Interpo- lation computes an integer value based on the input value, the desired range, the values of the smallest and largest of a set of values, and the probability distribution of the values in the set. Interpolation normally gives the statisti- cal mode of the location of a desired record in a random ordered file, that is, the most probable location of the record. Digitization scalar — sequence of scalars This operation transforms a scalar into a sequence of scalars. Numbering systems that allow the representation of integers as sequences of digits and strings as sequences of characters provide natural methods of digitization. Testing for equality value x value — boolean Rather than relying on multiway decisions to test two values for equality, a distinct operation is included in the basic set. Given two values of the same type (for example, two integers, two characters, two strings), this operation determines whether they are equal. Notice that the use of multiway branching plus equality testing closely matches the behaviour of most processors and programming languages which require two tests for a three-way branch (less than, equal, or greater than).BASIC CONCEPTS 2.2.2 Building procedures Building procedures are used to combine basic operations and simple algo- tithms to produce more complicated ones. In this section, we will define four building procedures: composition, alternation, conformation and self- organization. General references: (Darlington, J.. 78], [Barstow, D. [APOrIN ston, v., (Oj, poarsto J. et al., 80], [Merritt, S.M., oo SZ 2.2.2.1 Composition Composition is the main procedure for producing algorithms from atomic op- erations. Typically, but not exclusively, the composition of F; : Sx P — Rand Fo: Sx P — Rcan be expressed in a functional notation as F2(F,(S, P1), P2). A more general and hierarchical description of composition is that the descrip- tion of Fy uses F; instead of a basic operation. Although this definition is enough to include all types of composition, there are several common forms of composition that deserve to be identified explicitly. Divide and conquer This form uses a composition invoiving two algorithms for any problems that are greater than a critical size. The first algorithm splits a problem into (usually two) smaller problems. The composed algorithm is then recursively applied to each non-empty component, using recursion termination (see be- low) when appropriate. Finally the second algorithm is used to assemble the components’ results into one result. A typical example of divide and conquer is Quicksort (where the termination alternative may use a linear insertion sort). Diagrammatically: Divide and conquer solve~problem(A): if size(A) <= Critical—Size then End—Action else begin Split— problem; solve—problem(A1); solve—problem( Aa); ‘Assemble— Results a end; 17