Article

Free access

Stream processing

Authors:

Allen Goldberg,

Robert PaigeAuthors Info & Claims

LFP '84: Proceedings of the 1984 ACM Symposium on LISP and functional programming

Pages 53 - 62

https://doi.org/10.1145/800055.802021

Published: 06 August 1984 Publication History

Abstract

Stream processing is a basic method of code optimization related to loop fusion that can improve the space and speed of iterative applicative expressions by a process of loop combination. It has been studied before in applications to program improvement and batch oriented database restructuring. Previous attempts at implementation have been either highly restrictive, have required manual intervention, or have involved naive strategies resulting in impractically slow running times.

This paper defines a new model for a stream processing problem for handling a wide class of applicative expressions that can be evaluated by looping in an unordered way through a single set or tuple valued argument. This problem is formulated graph theoretically and shown to be NP-Hard, even in the presence of constraints. Two new efficient heuristic algorithms will be presented. The efficiency of these solutions allows stream processing to be applied dynamically as a database query optimization, and as an important component of a high level language optimizer.

Our method of stream processing has been implemented and used effectively within the RAPTS transformational programming system.

References

[1]

Allen. F. E., Cocke, John. A Catalogue of Optimizing Transformations. In Design and Optimization of Compilers, Randall Rustin, Ed. Prentice Hall. 1971, pp. 1-30.

[2]

Allen. J. R. Dependence Analysis for Subscripted Variables and its Application to Program Transformations. Ph.D. Th. Rice University. 1983.

Digital Library

[3]

Burge, William. An Optimizing Technique for High Level Programming Languages. Tech. Rept. Computer Science RC5834 #25271. IBM Research Center/Yorktown Heights, 1976.

[4]

Burstall. R. M., and Darlington. J. "A Transformation System for Developing Recursive Programs." JACM 24, 1 (Jan 1977).

Digital Library

[5]

Fagin, R., Nievergelt. J., Pippenger. N. and Strong, J. "Extendible Hashing - A Fast Access Method for Dynamic Files." ACM TODS 4. 3 (Sep 1979), 315-344.

Digital Library

[6]

Feather, Martin S. A System for Developing Programs by Transformation. Ph.D. Th., U. of Edinburgh, Dept. of Artificial Intelligence, 1979.

[7]

Friedman, D., Wise, D. CONS should not evaluate its arguments. In Automata, Languages, and Programming. S. Michaelson and R. Milner, Ed.Edinburgh Univ. Press, Edinburgh, 1976. pp. 257-284.

[8]

Friedman, D. and Wise, D. Aspects of Applicative Programming for File Systems. Proc. ACM Conf. on Language Design for Reliable Software. March, 1977, pp. 41-55. SIGPLAN Notices. Vol 12. Num. 3

Digital Library

[9]

Garey, Michael R., Johnson, David S. Computers and Intractability. Freeman. 1979.

[10]

Guibas, L. and Wyatt. K. Compilation and delayed evaluation in APL. Proceedings 5th ACM Symposium on Principles of Programming Languages, Jan, 1978, pp. 1-8.

Digital Library

[11]

Halmos, P.R. Naive Set Theory. Nan Nostrand, 1960.

[12]

Henderson, P. Functional Programming. Prentice-Hall, 1980.

Digital Library

[13]

Housel, B. "Pipelining: A Technique for Implementing Data Restructurers." TODS 4, 4 (Dec 1979), 470-492.

Digital Library

[14]

Kennedy, K. A Survey of Compiler Optimization Techniques. In Program Flow Analysis. Muchnick, S. and Jones, N., Eds., Prentice Hall. 1981, pp. 5-54.

[15]

Lu, Kang-Sen. Program Optimization Based on a Non-procedural Specification. Ph.D. Th., U. Penn., Dec 1981.

Digital Library

[16]

Morgenstern. Matthew. Automated Design and Optimization of Management Information System Software. Ph.D. Th. MIT. Laboratory for Computer Science, Sep 1976.

[17]

Overmars, M. The Design of Dynamic Data Structures. Springer-Verlag. Berlin. 1983. Lecture Notes in Computer Science #156

Digital Library

[18]

Paige. R. Formal Differentiation. UMI Research Press. Ann Arbor. Mich, 1981. Revision of Ph.D. thesis. NYU, June 1979

[19]

Paige. R. Transformational Programming — Applications to Algorithms and Systems. Proceedings Tenth ACM Symposium on Principles of Programming Languages, Jan, 1983, pp. 73-87.

Digital Library

[20]

Paige, Robert. Applications of Finite Differencing to Database Integrity Control and Query/Transaction Optimization. In Advances In Database Theory, Volume 2. Gallaire, H. Minker, J., and Nicolas. J.-M. Ed. Plenum Press, New York, 1984, pp. 171-210.

[21]

Paige, R. and Koenig. S. "Finite Differencing of Computable Expressions." ACM TOPLAS 4, 3 (July 1982). 402-454.

Digital Library

[22]

Prywes. N. Pnueli. A., and Shastry. S. "Use of a Non-Procedural Specification Language and Associated Program Generator in Software Development." ACM TOPLAS 1, 2 (Oct 1979). 196-217.

Digital Library

[23]

Reif, J. and Scherlis, W. Deriving Efficient Graph Algorithms. Carnegie-Mellon U., 1982. Technical Report

[24]

Schwartz, J. T. On Programming: An Interim Report on the SETL Project, Installments I and II. CIMS. New York Univ., New York. 1974.

[25]

Selinger, P.G., et.al. Access path selection in a relational database management system. SIGMOD, Boston, May, 1979. pp. 23-34.

Digital Library

[26]

Sharir, M. "Formal Integration - A Program Transformation Technique." Compout. Lang. 6, 1 (1982), 35-46.

[27]

Ullman, J.D. Principles of Database Systems. Computer Science Press, 1980.

Digital Library

[28]

Waters, R. Expressional Loops. Proceedings 11th ACM Symposium on Principles of Programming Languages, Jan. 1984.

Digital Library

[29]

Weixalbaum, E. Formal Integration. unpublished manuscript

Cited By

van Balen DKeller Gde Wolff IMcDonell TRainey MScholz S(2024)Fusing Gathers with Integer Linear ProgrammingProceedings of the 1st ACM SIGPLAN International Workshop on Functional Programming for Productivity and Performance10.1145/3677997.3678227(10-23)Online publication date: 28-Aug-2024
https://dl.acm.org/doi/10.1145/3677997.3678227
Shaikhha AKlonatos YParreaux LBrown LDashti MKoch CÖzcan FKoutrika GMadden S(2016)How to Architect a Query CompilerProceedings of the 2016 International Conference on Management of Data10.1145/2882903.2915244(1907-1922)Online publication date: 26-Jun-2016
https://dl.acm.org/doi/10.1145/2882903.2915244
Paige R(2008)A National Science Foundation ProposalAutomatic Program Development10.1007/978-1-4020-6585-9_2(7-27)Online publication date: 2008
https://doi.org/10.1007/978-1-4020-6585-9_2
Show More Cited By

Recommendations

Distributed stream join query processing with semijoins

This paper addresses the distributed stream processing of window-based multi-way join queries considering the semijoin as a key join operator. In distributed stream processing, data streams arriving at remote sites need to be shipped to the processing ...
Scalable and robust stream processing
Query processing of multi-way stream window joins

This paper introduces a class of join algorithms, termed W-join, for joining multiple infinite data streams. W-join addresses the infinite nature of the data streams by joining stream data items that lie within a sliding window and that match a certain ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LFP '84: Proceedings of the 1984 ACM Symposium on LISP and functional programming

August 1984

364 pages

ISBN:0897911423

DOI:10.1145/800055

Copyright © 1984 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 August 1984

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 30 of 109 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
450
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)10

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

van Balen DKeller Gde Wolff IMcDonell TRainey MScholz S(2024)Fusing Gathers with Integer Linear ProgrammingProceedings of the 1st ACM SIGPLAN International Workshop on Functional Programming for Productivity and Performance10.1145/3677997.3678227(10-23)Online publication date: 28-Aug-2024
https://dl.acm.org/doi/10.1145/3677997.3678227
Shaikhha AKlonatos YParreaux LBrown LDashti MKoch CÖzcan FKoutrika GMadden S(2016)How to Architect a Query CompilerProceedings of the 2016 International Conference on Management of Data10.1145/2882903.2915244(1907-1922)Online publication date: 26-Jun-2016
https://dl.acm.org/doi/10.1145/2882903.2915244
Paige R(2008)A National Science Foundation ProposalAutomatic Program Development10.1007/978-1-4020-6585-9_2(7-27)Online publication date: 2008
https://doi.org/10.1007/978-1-4020-6585-9_2
Paige R(2005)An NSF ProposalHigher-Order and Symbolic Computation10.1007/s10990-005-7009-218:1-2(211-235)Online publication date: 1-Jun-2005
https://dl.acm.org/doi/10.1007/s10990-005-7009-2
Paige R(2005)Viewing a program transformation system at workProgramming Language Implementation and Logic Programming10.1007/3-540-58402-1_3(5-24)Online publication date: 28-May-2005
https://doi.org/10.1007/3-540-58402-1_3
Kennedy KMcKinley K(2005)Maximizing loop parallelism and improving data locality via loop fusion and distributionLanguages and Compilers for Parallel Computing10.1007/3-540-57659-2_18(301-320)Online publication date: 31-May-2005
https://doi.org/10.1007/3-540-57659-2_18
Gao GOlsen RSarkar VThekkath R(2005)Collective loop fusion for array contractionLanguages and Compilers for Parallel Computing10.1007/3-540-57502-2_53(281-295)Online publication date: 3-Jun-2005
https://doi.org/10.1007/3-540-57502-2_53
Paige RHenglein F(2005)Mechanical translation of set theoretic problem specifications into efficient RAM code — A case studyEUROCAL '8510.1007/3-540-15984-3_329(554-567)Online publication date: 8-Jun-2005
https://doi.org/10.1007/3-540-15984-3_329
Hu ZYokoyama TTakeichi M(2005)Program optimizations and transformations in calculation formProceedings of the 2005 international conference on Generative and Transformational Techniques in Software Engineering10.1007/11877028_5(144-168)Online publication date: 4-Jul-2005
https://dl.acm.org/doi/10.1007/11877028_5
Kennedy K(2001)Fast Greedy Weighted FusionInternational Journal of Parallel Programming10.1023/A:101224183076229:5(463-491)Online publication date: 1-Oct-2001
https://dl.acm.org/doi/10.1023/A%3A1012241830762
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents