chapter

Design and Engineering of External Memory Traversal Algorithms for General Graphs

Authors:

Ulrich MeyerAuthors Info & Claims

Algorithmics of Large and Complex Networks: Design, Analysis, and Simulation

June 2009

Pages 1 - 33

https://doi.org/10.1007/978-3-642-02094-0_1

Published: 29 June 2009 Publication History

Abstract

Large graphs arise naturally in many real world applications. The actual performance of simple RAM model algorithms for traversing these graphs (stored in external memory) deviates significantly from their linear or near-linear predicted performance because of the large number of I/Os they incur. In order to alleviate the I/O bottleneck, many external memory graph traversal algorithms have been designed with provable worst-case guarantees. We describe the various techniques used in the design and engineering of such algorithms and survey the state-of-the-art in I/O-efficient graph traversal algorithms.

References

[1]

Abello, J., Buchsbaum, A., Westbrook, J.: A functional approach to external graph algorithms. Algorithmica 32(3), 437-458 (2002).

[2]

Abello, J., Pardalos, P., Resende, M.: On maximum clique problems in very large graphs. External Memory Algorithms, AMS-DIMACS Series in Discrete Mathematics and Theoretical Computer Science 50, 119-130 (1999).

[3]

Aggarwal, A., Vitter, J.S.: The input/output complexity of sorting and related problems. Communications of the ACM 31(9), 1116-1127 (1988).

[4]

Ajwani, D., Dementiev, R., Meyer, U.: A computational study of external memory bfs algorithms. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 601-610 (2006).

Digital Library

[5]

Ajwani, D., Meyer, U., Osipov, V.: Improved external memory BFS implementation. In: Proceedings of the workshop on Algorithm Engineering and Experiments (ALENEX), pp. 3-12 (2007).

[6]

Allulli, L., Lichodzijewski, P., Zeh, N.: A faster cache-oblivious shortest-path algorithm for undirected graphs with bounded edge lengths. In: Proceedings of the 18th annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 910-919 (2007).

[7]

Arge, L.: The Buffer Tree: A new technique for optimal I/O-algorithms. In: Sack, J.-R., Akl, S.G., Dehne, F., Santoro, N. (eds.) WADS 1995. LNCS, vol. 955, pp. 334-345. Springer, Heidelberg (1995).

[8]

Arge, L., Bender, M.A., Demaine, E.D., Holland-Minkley, B., Munro, J.I.: Cache-oblivious priority queue and graph algorithm applications. In: Proceedings of the 34th Annual ACM Symposium on Theory of Computing (STOC), pp. 268-276 (2002).

[9]

Arge, L., Brodal, G., Toma, L.: On external-memory MST, SSSP and multiway planar graph separation. In: Halldórsson, M.M. (ed.) SWAT 2000. LNCS, vol. 1851, pp. 433-447. Springer, Heidelberg (2000).

Digital Library

[10]

Arge, L., Chase, J.S., Halpin, P., Toma, L., Vitter, J.S., Urban, D., Wickremesinghe, R.: Efficient flow computation on massive grid terrain datasets. Geoinformatica 7(4), 283-313 (2003).

[11]

Arge, L., Meyer, U., Toma, L.: External memory algorithms for diameter and all-pairs shortest-paths on sparse graphs. In: Díaz, J., Karhumäki, J., Lepistö, A., Sannella, D. (eds.) ICALP 2004. LNCS, vol. 3142, pp. 146-157. Springer, Heidelberg (2004).

[12]

Arge, L., Procopiuc, O., Vitter, J.: Implementing I/O-efficient data structures using TPIE. In: Möhring, R.H., Raman, R. (eds.) ESA 2002. LNCS, vol. 2461, pp. 88- 100. Springer, Heidelberg (2002).

Digital Library

[13]

Boruvka, O.: O jistém problému minimálním. In: Práce, Moravské Prirodovedecké Spolecnosti, pp. 1-58 (1926).

[14]

Brandes, U., Erlebach, T. (eds.): Network Analysis. LNCS, vol. 3418. Springer, Heidelberg (2005).

[15]

Brodal, G., Fagerberg, R.: Cache oblivious distribution sweeping. In: Widmayer, P., Triguero, F., Morales, R., Hennessy, M., Eidenbenz, S., Conejo, R. (eds.) ICALP 2002. LNCS, vol. 2380, pp. 426-438. Springer, Heidelberg (2002).

Digital Library

[16]

Brodal, G., Fagerberg, R., Meyer, U., Zeh, N.: Cache-oblivious data structures and algorithms for undirected breadth-first search and shortest paths. In: Hagerup, T., Katajainen, J. (eds.) SWAT 2004. LNCS, vol. 3111, pp. 480-492. Springer, Heidelberg (2004).

[17]

Brodal, G.S., Fagerberg, R.: Funnel heap - a cache oblivious priority queue. In: Bose, P., Morin, P. (eds.) ISAAC 2002. LNCS, vol. 2518, pp. 219-228. Springer, Heidelberg (2002).

Digital Library

[18]

Brodal, G.S., Fagerberg, R., Vinther, K.: Engineering a cache-oblivious sorting algorithm. In: Proceedings of the sixth workshop on Algorithm Engineering and Experiments (ALENEX), pp. 4-17 (2004).

[19]

Buchsbaum, A., Goldwasser, M., Venkatasubramanian, S., Westbrook, J.: On external memory graph traversal. In: 11th Annual Symposium on Discrete Algorithms (SODA), pp. 859-860. ACM-SIAM (2000).

[20]

Chan, T.M.: All-pairs shortest paths with real weights in O(n ³ log n) time. Algorithmica 50(2), 236-243 (2008).

[21]

Chazelle, B.: A minimum spanning tree algorithm with inverse-ackermann type complexity. JACM 47(6), 1028-1047 (2000).

[22]

Chen, M., Chowdhury, R.A., Ramachandran, V., Roche, D.L., Tong, L.: Priority queues and dijkstra's algorithm. Technical report TR-07-54, The University of Texas at Austin, Department of Computer Sciences (2007).

[23]

Chiang, Y.J., Goodrich, M.T., Grove, E.F., Tamasia, R., Vengroff, D.E., Vitter, J.S.: External memory graph algorithms. In: Proceedings of the 6th annual Symposium on Discrete Algorithms (SODA), pp. 139-149. ACM-SIAM (1995).

[24]

Chowdhury, R.A., Ramachandran, V.: External-memory exact and approximate all-pairs shortest-paths in undirected graphs. In: Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 735-744 (2005).

[25]

Cormen, T., Leiserson, C., Rivest, R.: Introduction to Algorithms. The MIT Press, Cambridge (1989).

[26]

de Kunder, M.: World wide web size, http://www.worldwidewebsize.com/

[27]

Dementiev, R., Kettner, L., Sanders, P.: Stxxl: Standard Template Library for XXL Data Sets. Technical Report 18, Fakultät für Informatik, University of Karlsruhe (2005).

[28]

Dementiev, R., Kettner, L., Sanders, P.: STXXL: Standard Template library for XXL data sets. Software: Practice and Experience 38(6), 589-637 (2008).

[29]

Dementiev, R., Sanders, P., Kettner, L.: STXXL: Standard Template library for XXL data sets. In: Brodal, G.S., Leonardi, S. (eds.) ESA 2005. LNCS, vol. 3669, pp. 640-651. Springer, Heidelberg (2005).

[30]

Dementiev, R., Sanders, P., Schultes, D., Sibeyn, J.: Engineering an external memory minimum spanning tree algorithm. In: Proceedings of the 3rd International Conference on Theoretical Computer Science (TCS), pp. 195-208. Kluwer, Dordrecht (2004).

[31]

Dijkstra, E.W.: A note on two problems in connexion with graphs. In Numerische Mathematik 1, 269-271 (1959).

[32]

Edelkamp, S., Jabbar, S.: Large-scale directed model checking LTL. In: Valmari, A. (ed.) SPIN 2006. LNCS, vol. 3925, pp. 1-18. Springer, Heidelberg (2006).

Digital Library

[33]

Edelkamp, S., Jabbar, S., Schrödl, S.: External A*. In: Biundo, S., Frühwirth, T., Palm, G. (eds.) KI 2004. LNCS (LNAI), vol. 3238, pp. 226-240. Springer, Heidelberg (2004).

[34]

Erdös, P., Rényi, A.: On random graphs. Publ. Math. Debrecen 6, 290-297 (1959).

[35]

Erdös, P., Rényi, A.: On the evolution of random graphs. Magyar Tud. Akad. Mat. Kutato Int. Kozl. 5, 17-61 (1960).

[36]

Fredman, M.L., Tarjan, R.E.: Fibonacci heaps and their uses in improved network optimization algorithms. Journal of the ACM 34(3), 596-615 (1987).

[37]

Frigo, M., Leiserson, C.E., Prokop, H., Ramachandran, S.: Cache-oblivious algorithms. In: 40th Annual Symposium on Foundations of Computer Science, pp. 285-297. IEEE Computer Society Press, Los Alamitos (1999).

Digital Library

[38]

Galler, B.A., Fisher, M.J.: An improved equivalence algorithm. Communications of the ACM 7(5), 301-303 (1964).

[39]

Goldberg, A., Werneck, R.: Computing point-to-point shortest paths from external memory. In: Proceedings of the 7th workshop on Algorithms Engineering and Experiments (ALENEX), pp. 26-40 (2005).

[40]

Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics SSC 4(2), 100-107 (1968).

[41]

Jabbar, S., Edelkamp, S.: I/O efficient directed model checking. In: Cousot, R. (ed.) VMCAI 2005. LNCS, vol. 3385, pp. 313-329. Springer, Heidelberg (2005).

[42]

Jabbar, S., Edelkamp, S.: Parallel external directed model checking with linear I/O. In: Emerson, E.A., Namjoshi, K.S. (eds.) VMCAI 2006. LNCS, vol. 3855, pp. 237-251. Springer, Heidelberg (2005).

[43]

Jarník, V.: O jistém problému minimálním {About a certain minimal problem}. Práce Moravské Přírodovedecké Společnosti 6, 57-63 (1930).

[44]

Karger, D., Klein, P., Tarjan, R.: A randomized linear time algorithm to find minimum spanning trees. Journal of the ACM 42(2), 321-328 (1995).

[45]

Kruskal, J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society 7(1), 48-50 (1956).

[46]

Kumar, V., Schwabe, E.J.: Improved algorithms and data structures for solving graph problems in external memory. In: Proceedings of the 8th Symposium on Parallel and Distributed Processing, pp. 169-177. IEEE, Los Alamitos (1996).

[47]

Laura, L., Leonardi, S., Millozzi, S., Meyer, U., Sibeyn, J.F.: Algorithms and experiments for the webgraph. In: Di Battista, G., Zwick, U. (eds.) ESA 2003. LNCS, vol. 2832, pp. 703-714. Springer, Heidelberg (2003).

[48]

Maon, Y., Scheiber, B., Vishkin, U.: Parallel ear decomposition search (EDS) and st-numbering in graphs. Theoretical Computer Science 47, 277-298 (1986).

[49]

Mehlhorn, K., Meyer, U.: External-memory Breadth-First Search with sublinear I/O. In: Möhring, R.H., Raman, R. (eds.) ESA 2002. LNCS, vol. 2461, pp. 723-735. Springer, Heidelberg (2002).

Digital Library

[50]

Mehlhorn, K., Naher, S.: The LEDA Platform of Combinatorial and Geometric Computing. Cambridge University Press, Cambridge (1999).

[51]

Meyer, U., Zeh, N.: I/O-efficient undirected shortest paths. In: Di Battista, G., Zwick, U. (eds.) ESA 2003. LNCS, vol. 2832, pp. 434-445. Springer, Heidelberg (2003).

[52]

Meyer, U., Zeh, N.: I/O-efficient undirected shortest paths with unbounded edge lengths. In: Azar, Y., Erlebach, T. (eds.) ESA 2006. LNCS, vol. 4168, pp. 540-551. Springer, Heidelberg (2006).

[53]

Munagala, K., Ranade, A.: I/O-complexity of graph algorithms. In: Proceedings of the 10th Annual Symposium on Discrete Algorithms (SODA), pp. 687-694. ACM-SIAM (1999).

[54]

Pettie, S.: A new approach to all-pairs shortest paths on real-weighted graphs. Theoretical Computer Science 312(1), 47-74 (2004).

[55]

Pettie, S., Ramachandran, V.: An optimal minimum spanning tree algorithm. Journal of the ACM 49(1), 16-34 (2002).

[56]

Prim, R.C.: Shortest connection networks and some generalisations. Bell System Technical Journal 36, 1389-1401 (1957).

[57]

Sach, B., Clifford, R.: An empirical study of cache-oblivious priority queues and their application to the shortest path problem (2008), http://arxiv.org/abs/0802.1026v1

[58]

Sanders, P.: Random permutations on distributed, external and hierarchical memory. Information Processing Letters 67, 305-309 (1998).

[59]

Sanders, P., Schultes, D., Vetter, C.: Mobile route planning. In: Halperin, D., Mehlhorn, K. (eds.) Esa 2008. LNCS, vol. 5193, pp. 732-743. Springer, Heidelberg (2008).

[60]

Sibeyn, J.F.: From parallel to external list ranking, Technical report, Max Planck Institut für Informatik, Saarbrücken, Germany (1997).

[61]

Sibeyn, J.F., Abello, J., Meyer, U.: Heuristics for semi-external depth first search on directed graphs. In: Proceedings of the 14th Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA), pp. 282-292 (2002).

[62]

Stepanov, A., Lee, M.: The Standard Template Library. Hewlett Packard Laboratories (1995), http://www.sgi.com/tech/stl/

[63]

Tarjan, R.E.: Efficiency of a good but not linear set union algorithm. Journal of ACM 22(2), 215-225 (1975).

[64]

Seagate Technology, http://www.seagate.com/cda/products/discsales/ marketing/detail/0,1081,628,00.html

Cited By

Gao TLi JMa H(2023)A New Approach for Semi-External Topological Sorting on Big GraphsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327452835:12(12430-12443)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3274528
Meyer UTran HTsakalidis K(2023)Certifying Induced Subgraphs in Large GraphsWALCOM: Algorithms and Computation10.1007/978-3-031-27051-2_20(229-241)Online publication date: 22-Mar-2023
https://dl.acm.org/doi/10.1007/978-3-031-27051-2_20
Zhang ZYu JQin LShang ZSellis TDavidson SIves Z(2015)Divide & ConquerProceedings of the 2015 ACM SIGMOD International Conference on Management of Data10.1145/2723372.2723740(445-458)Online publication date: 27-May-2015
https://dl.acm.org/doi/10.1145/2723372.2723740
Show More Cited By

Index Terms

Design and Engineering of External Memory Traversal Algorithms for General Graphs
1. Mathematics of computing
  1. Discrete mathematics
    1. Graph theory

Index terms have been assigned to the content through auto-classification.

Recommendations

External memory algorithms for path traversal in graphs
External Memory Algorithms for Outerplanar Graphs
ISAAC '99: Proceedings of the 10th International Symposium on Algorithms and Computation

We present external memory algorithms for outerplanarity testing, embedding outerplanar graphs, breadth-first search (BFS) and depth-first search (DFS) in outerplanar graphs, and finding a 2/3-separator of size 2 for a given outerplanar graph. Our ...
External Memory Algorithms for String Problems
Workshop on Combinatorial Algorithms

In this paper we present external memory algorithms for some string problems. External memory algorithms have been developed in many research areas, as the speed gap between fast internal memory and slow external memory continues to grow. The goal of ...

Comments

Information & Contributors

Information

Published In

cover image Guide books

Algorithmics of Large and Complex Networks: Design, Analysis, and Simulation

June 2009

400 pages

ISBN:9783642020933

Editors:
Jürgen Lerner
Department of Computer & Information Science, University of Konstanz, Konstanz, Germany 78457
,
Dorothea Wagner
Institute of Theoretical Informatics, Technical University of Karlsruhe, Karlsruhe, Germany 76128
,
Katharina A. Zweig
Department of Biological Physics, Eötvös Lorand University, Budapest, Hungary 1117

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 29 June 2009

Qualifiers

Chapter

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao TLi JMa H(2023)A New Approach for Semi-External Topological Sorting on Big GraphsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327452835:12(12430-12443)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3274528
Meyer UTran HTsakalidis K(2023)Certifying Induced Subgraphs in Large GraphsWALCOM: Algorithms and Computation10.1007/978-3-031-27051-2_20(229-241)Online publication date: 22-Mar-2023
https://dl.acm.org/doi/10.1007/978-3-031-27051-2_20
Zhang ZYu JQin LShang ZSellis TDavidson SIves Z(2015)Divide & ConquerProceedings of the 2015 ACM SIGMOD International Conference on Management of Data10.1145/2723372.2723740(445-458)Online publication date: 27-May-2015
https://dl.acm.org/doi/10.1145/2723372.2723740
Zhang ZYu JQin LChang LLin X(2015)I/O efficientThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-014-0372-z24:2(245-270)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.1007/s00778-014-0372-z
Zhang ZYu JQin LChang LLin XRoss KSrivastava DPapadias D(2013)I/O efficientProceedings of the 2013 ACM SIGMOD International Conference on Management of Data10.1145/2463676.2463703(181-192)Online publication date: 22-Jun-2013
https://dl.acm.org/doi/10.1145/2463676.2463703
Buluç AMadduri KLathrop SCosta JKramer W(2011)Parallel breadth-first search on distributed memory systemsProceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/2063384.2063471(1-12)Online publication date: 12-Nov-2011
https://dl.acm.org/doi/10.1145/2063384.2063471
Pearce RGokhale MAmato NHess B(2010)Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External MemoryProceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2010.34(1-11)Online publication date: 13-Nov-2010
https://dl.acm.org/doi/10.1109/SC.2010.34

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Chapter

Media

Figures

Other

Tables

View Table of Contents