Methods in

Algorithmic Analysis
 CHAPMAN & HALL/CRC
COMPUTER and INFORMATION SCIENCE SERIES

Series Editor: Sartaj Sahni

PUBLISHED TITLES
ADVERSARIAL REASONING: COMPUTATIONAL APPROACHES HANDBOOK OF PARALLEL COMPUTING: MODELS,
TO READING THE OPPONENT’S MIND ALGORITHMS AND APPLICATIONS
Alexander Kott and William M. McEneaney Sanguthevar Rajasekaran and John Reif
DISTRIBUTED SENSOR NETWORKS HANDBOOK OF REAL-TIME AND EMBEDDED SYSTEMS
S. Sitharama Iyengar and Richard R. Brooks Insup Lee, Joseph Y-T. Leung, and Sang H. Son
DISTRIBUTED SYSTEMS: AN ALGORITHMIC APPROACH HANDBOOK OF SCHEDULING: ALGORITHMS, MODELS,
Sukumar Ghosh AND PERFORMANCE ANALYSIS
Joseph Y.-T. Leung
ENERGY EFFICIENT HARDWARE-SOFTWARE
CO-SYNTHESIS USING RECONFIGURABLE HARDWARE HIGH PERFORMANCE COMPUTING IN REMOTE SENSING
Jingzhao Ou and Viktor K. Prasanna Antonio J. Plaza and Chein-I Chang
FUNDEMENTALS OF NATURAL COMPUTING: BASIC INTRODUCTION TO NETWORK SECURITY
CONCEPTS, ALGORITHMS, AND APPLICATIONS Douglas Jacobson
Leandro Nunes de Castro
METHODS IN ALGORITHMIC ANALYSIS
HANDBOOK OF ALGORITHMS FOR WIRELESS Vladimir A. Dobrushkin
NETWORKING AND MOBILE COMPUTING
Azzedine Boukerche PERFORMANCE ANALYSIS OF QUEUING AND COMPUTER
NETWORKS
HANDBOOK OF APPROXIMATION ALGORITHMS G. R. Dattatreya
AND METAHEURISTICS
Teofilo F. Gonzalez THE PRACTICAL HANDBOOK OF INTERNET COMPUTING
Munindar P. Singh
HANDBOOK OF BIOINSPIRED ALGORITHMS
AND APPLICATIONS SCALABLE AND SECURE INTERNET SERVICES AND
Stephan Olariu and Albert Y. Zomaya ARCHITECTURE
Cheng-Zhong Xu
HANDBOOK OF COMPUTATIONAL MOLECULAR BIOLOGY
Srinivas Aluru SPECULATIVE EXECUTION IN HIGH PERFORMANCE
COMPUTER ARCHITECTURES
HANDBOOK OF DATA STRUCTURES AND APPLICATIONS David Kaeli and Pen-Chung Yew
Dinesh P. Mehta and Sartaj Sahni
VEHICULAR NETWORKS: FROM THEORY TO PRACTICE
HANDBOOK OF DYNAMIC SYSTEM MODELING Stephan Olariu and Michele C. Weigle
Paul A. Fishwick
 Methods in
Algorithmic Analysis

Vladimir a. dobrushkin
brown uniVersity
ProVidence, rhode island, u.s.a.
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2009 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Version Date: 20131120

International Standard Book Number-13: 978-1-4200-6830-6 (eBook - PDF)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti-
lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy-
ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the
publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents

Preface xiii

List of Symbols xvii

Abbreviations xix

1 Preliminaries 1
1.1 Why Do We Analyze Algorithms? . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Cost of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 One Problem – Several Solutions . . . . . . . . . . . . . . . . . . . 3
1.1.3 Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 Proof by Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.2 Induction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3 Iteration and Recursion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2 Combinatorics 25
2.1 Properties of Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.1.1 Index Transformations . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2 Multiple Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.2.1 Changing Order of Summation . . . . . . . . . . . . . . . . . . . . . 34
2.2.2 Summations and Finite Differences . . . . . . . . . . . . . . . . . . 36
2.2.3 Summation by Parts . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.3 Principles of Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.4 Permutations and Combinations . . . . . . . . . . . . . . . . . . . . . . . . 44
2.4.1 Combinations and Lattice Paths . . . . . . . . . . . . . . . . . . . . 48
2.5 Binomial Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
 2.5.1 Definitions and Properties . . . . . . . . . . . . . . . . . . . . . . . 56
2.5.2 Transformations and Basic Sums . . . . . . . . . . . . . . . . . . . . 58
2.5.3 Inverse Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
2.5.4 Vandermonde Convolution . . . . . . . . . . . . . . . . . . . . . . . 66
2.6 Binomial Coefficients and Hypergeometric Functions . . . . . . . . . . . . . 72
2.6.1 Abel’s Identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
2.7 Stirling Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

3 Probability 79
3.1 Set Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.2 Sample Space and Random Variables . . . . . . . . . . . . . . . . . . . . . . 82
3.3 Calculating Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.4 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.4.1 Probability Mass Function . . . . . . . . . . . . . . . . . . . . . . . 94
3.4.2 Expected Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.4.3 Variance and Moments . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.4.4 Functions of Random Variables . . . . . . . . . . . . . . . . . . . . 119
3.5 Conditional Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
3.6 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
3.7 Joint Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
3.8 Dependent Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 130

4 More about Probability 135

4.1 Special Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.1.1 Bernoulli Variables and the Binomial Distribution . . . . . . . . . . . 136
4.1.2 The Multinomial Distribution . . . . . . . . . . . . . . . . . . . . . 140
4.1.3 The Geometric Distribution . . . . . . . . . . . . . . . . . . . . . . 141
4.1.4 The Negative-Binomial Distribution . . . . . . . . . . . . . . . . . . 148
4.1.5 The Hypergeometric Distribution . . . . . . . . . . . . . . . . . . . 151
4.1.6 The Poisson Distribution . . . . . . . . . . . . . . . . . . . . . . . . 154
4.1.7 The Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . 157
4.2 Types of Probabilistic Convergence . . . . . . . . . . . . . . . . . . . . . . 158
4.3 The Theorem of Total Probability . . . . . . . . . . . . . . . . . . . . . . . 162
4.4 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
4.5 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
 4.6 Order Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
4.7 Chebyshev Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
4.8 Sundry Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

5 Recurrences or Difference Equations 199

5.1 How Do Difference Equations Arise? . . . . . . . . . . . . . . . . . . . . . 200
5.2 Properties of Difference Equations . . . . . . . . . . . . . . . . . . . . . . . 208
5.3 First Order Linear Difference Equations . . . . . . . . . . . . . . . . . . . . 215
5.4 Recurrences with “Integer Functions” . . . . . . . . . . . . . . . . . . . . . 221
5.4.1 Divide-and-Conquer Recurrences . . . . . . . . . . . . . . . . . . . 221
5.4.2 Binary Search Recurrence . . . . . . . . . . . . . . . . . . . . . . . 224
5.4.3 Mergesort Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . 227
5.5 Quicksort Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
5.5.1 Some Full-History Recurrences . . . . . . . . . . . . . . . . . . . . 233
5.6 Recurrences in Numerical Analysis . . . . . . . . . . . . . . . . . . . . . . . 236
5.7 Continued Fractions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
5.8 Partial Difference Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 251
5.8.1 The Relations with Continuous Calculations . . . . . . . . . . . . . . 254
5.8.2 Counting Arrangements—with and without Repetition . . . . . . . . 256
5.8.3 Stirling Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
5.9 Some Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
5.9.1 Bounds from Recurrences . . . . . . . . . . . . . . . . . . . . . . . 263
5.9.2 Recurrences and Finite Differences . . . . . . . . . . . . . . . . . . 266

6 Introduction to Generating Functions 271

6.1 Generating Functions — Definitions . . . . . . . . . . . . . . . . . . . . . . 272
6.1.1 Ordinary Generating Functions . . . . . . . . . . . . . . . . . . . . . 272
6.1.2 Exponential Generating Functions . . . . . . . . . . . . . . . . . . . 277
6.1.3 Some Generalizations . . . . . . . . . . . . . . . . . . . . . . . . . 280
6.1.4 Multivariate Generating Functions . . . . . . . . . . . . . . . . . . . 283
6.1.5 Multisection of Series . . . . . . . . . . . . . . . . . . . . . . . . . 285
6.2 Extraction of Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
6.2.1 Transformations between the Generating Functions . . . . . . . . . . 293
6.2.2 Multivariate Generating Functions . . . . . . . . . . . . . . . . . . . 294
6.2.3 Recurrences from Generating Functions . . . . . . . . . . . . . . . . 296
 6.3 Counting Binary Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
6.4 Solving Recurrences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
6.4.1 Ordinary Recurrence Relations . . . . . . . . . . . . . . . . . . . . . 305
6.4.2 Vector Recurrence Relations . . . . . . . . . . . . . . . . . . . . . . 315
6.4.3 Partial Difference Equations . . . . . . . . . . . . . . . . . . . . . . 319
6.4.4 Walks on the Integer Grid . . . . . . . . . . . . . . . . . . . . . . . 323
6.5 Snake Oil Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
6.6 Applications in Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
6.6.1 Definition of Generating Functions Used in Probability . . . . . . . . 331
6.6.2 Examples and Problems . . . . . . . . . . . . . . . . . . . . . . . . 335
6.6.3 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
6.6.4 Quicksort and Binary Search Analysis . . . . . . . . . . . . . . . . . 340
6.7 The Lagrange Inversion Theorem . . . . . . . . . . . . . . . . . . . . . . . . 347

7 Enumeration with Generating Functions 355

7.1 Definition of Enumerators . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
7.2 Sum and Product Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
7.2.1 The Sum Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
7.2.2 The Product Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
7.3 Counting Compositions of Integers . . . . . . . . . . . . . . . . . . . . . . . 364
7.3.1 Homogeneous Compositions . . . . . . . . . . . . . . . . . . . . . . 364
7.3.2 Inhomogeneous Compositions . . . . . . . . . . . . . . . . . . . . . 367
7.3.3 Compositions with Restrictions . . . . . . . . . . . . . . . . . . . . 369
7.3.4 Heterogeneous Components . . . . . . . . . . . . . . . . . . . . . . 372
7.3.5 Limited Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
7.4 Further Set Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
7.4.1 Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
7.4.2 Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
7.4.3 Power Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390
7.4.4 Multiset Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
7.5 Partitions of Integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
7.6 Exponential Enumerators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
7.6.1 The Sum and Product of Labeled Structures . . . . . . . . . . . . . . 404
7.6.2 Permutations and Cycles . . . . . . . . . . . . . . . . . . . . . . . . 408
 7.6.3 Shuffle Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
7.6.4 The Birthday Problems . . . . . . . . . . . . . . . . . . . . . . . . . 416

8 Further Enumeration Methods 423

8.1 Enumeration of Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
8.1.1 Unlabeled Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
8.1.2 Labeled Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
8.1.3 Counting Alternating Permutations . . . . . . . . . . . . . . . . . . 430
8.2 Occupancy Enumeration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
8.2.1 Distribution of Identical Balls into Distinguishable Bins . . . . . . . 435
8.2.2 Distribution of Distinct Objects into Ordered Cells . . . . . . . . . . 439
8.2.3 Distribution of Identical Objects into Identical Cells . . . . . . . . . 445
8.2.4 Distribution of Distinct Objects into Identical Cells . . . . . . . . . . 445
8.3 The Principle of Inclusion and Exclusion (PIE) . . . . . . . . . . . . . . . . . 446
8.3.1 The PIE for Homogeneous Properties . . . . . . . . . . . . . . . . . 455
8.4 Extensions and Further Applications of the PIE . . . . . . . . . . . . . . . . 460
8.4.1 The PIE via the Symbolic Method . . . . . . . . . . . . . . . . . . . 464
8.5 Probabilistic Inclusion – Exclusion Principle . . . . . . . . . . . . . . . . . . 468
8.6 Runs in Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
8.6.1 Counting Permutations of [1..n] with k Ascents . . . . . . . . . . . . 480
8.6.2 Counting Permutations of [1..n] with Runs of Ascents of Length r . . 482
8.6.3 Counting Permutations of [1..n] with m Maximal Runs of Ascents . . 483
8.7 Special Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483

9 Combinatorics of Strings 489

9.1 Operations on Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
9.2 Regular Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
9.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
9.2.2 Finite State Automata . . . . . . . . . . . . . . . . . . . . . . . . . 497
9.2.3 Finite State Automata and Regular Languages . . . . . . . . . . . . . 499
9.3 Counting Regular Languages . . . . . . . . . . . . . . . . . . . . . . . . . . 503
9.3.1 Word Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
9.3.2 Counting Regular Languages . . . . . . . . . . . . . . . . . . . . . . 505
9.3.3 Admissibility Considerations . . . . . . . . . . . . . . . . . . . . . . 514
9.4 Waiting Time Probabilistic Problems . . . . . . . . . . . . . . . . . . . . . . 517