Compositional structures for streaming applications
Pages 352 - 361
Abstract
This paper describes an ongoing project to develop a Python software package, IoTPy, that helps beginning programmers build modular applications that process streams of data collected from sensors, social media and other sources, and to reason about the correctness of their applications in a compositional fashion. IoTPy helps build streaming applications in four ways: (1) enables the construction of non-terminating applications that continuously process endless streams of data by encapsulating terminating programs; (2) supports computation throughout a network of nodes from sensors at the edges of the network to the cloud and back to actuators at the edge; (3) allows users to separate concerns of the logic of an application from the parallel hardware on which the application runs, and (4) supports proofs and testing of the correct behavior of a composition from the specifications of its components.
References
[1]
Darko Anicic, Paul Fodor, Sebastian Rudolph, and Nenad Stojanovic. 2011. EPSPARQL: A Unified Language for Event Processing and Stream Reasoning. In Proceedings of the 20th International Conference on World Wide Web (WWW '11). ACM, New York, NY, USA, 635--644.
[2]
David A. Anisi. 2003. Optimal Motion Control of a Ground Vehicle. Master's thesis. Royal Institute of Technology (KTH), Stockholm, Sweden.
[3]
K. Mani Chandy and Carl Kesselman. 1993. Research Directions in Concurrent Object-oriented Programming. MIT Press, Cambridge, MA, USA, Chapter CC++: A Declarative Concurrent Object-oriented Programming Notation, 281--313. http://dl.acm.org/citation.cfm?id=167978.167993
[4]
K. Mani Chandy and Jayadev Misra. 1988. Parallel Program Design: A Foundation. Addison-Wesley.
[5]
K. Mani Chandy, Concetta Pilotto, and Ryan McLean. 2008. Networked Sensing Systems for Detecting People Carrying Radioactive Material. In Proceedings of the Fifth International Conference Networked Sensing Systems (INSS'08). IEEE.
[6]
K. Mani Chandy and Roy W. Schulte. 2010. Event Processing: Designing IT Systems for Agile Companies. McGraw Hill.
[7]
K. Mani Chandy and Stephen Taylor. 1992. An Introduction to Parallel Programming. Jones and Bartlett Publishers, Inc., USA.
[8]
Robert W. Clayton, Thomas Heaton, Monica Kohler, K. Mani. Chandy, Richard Guy, and Julian Bunn. 2015. Community Seismic Network: A Dense Array to Sense Earthquake Strong Motion. Seismological Research Letters 86, 5 (April 2015).
[9]
Opher Etzion and Peter Niblett. 2011. Event Processing in Action. Manning Publications.
[10]
Matthew Faulkner, Robert Clayton, Thomas Heaton, K. Mani Chandy, Monica Kohler, Julian Bunn, Richard Guy, Annie Liu, Michael Olson, MingHei Cheng, and Andreas Krause. 2014. Community Sense and Response Systems: Your Phone As Quake Detector. Commun. ACM 57, 7 (July 2014), 66--75.
[11]
I.T. Foster and K.M. Chandy. 1995. FORTRAN M - A Language for Modular Parallel Programming. J. Parallel Distrib. Comput. 26, 1 (April 1995), 24--35.
[12]
Ian Foster and Stephen Taylor. 1990. Strand: New Concepts in Parallel Programming. Prentice-Hall, Inc., Upper Saddle River, NJ, USA.
[13]
Namit Jain, Shailendra Mishra, Anand Srinivasan, Johannes Gehrke, Jennifer Widom, Hari Balakrishnan, Uĝur Çetintemel, Mitch Cherniack, Richard Tibbetts, and Stan Zdonik. 2008. Towards a Streaming SQL Standard. Proc. VLDB Endow. 1, 2 (Aug. 2008), 1379--1390.
[14]
David Kitchin, Adrian Quark, William Cook, and Jayadev Misra. 2009. The Orc Programming Language. In Proceedings of the Joint 11th IFIP WG 6.1 International Conference FMOODS '09 and 29th IFIP WG 6.1 International Conference FORTE '09 on Formal Techniques for Distributed Systems (FMOODS '09/FORTE '09). Springer-Verlag, Berlin, Heidelberg, 1--25.
[15]
Leslie Lamport. 1994. The Temporal Logic of Actions. ACM Trans. Program. Lang. Syst. 16, 3 (May 1994), 872--923.
[16]
Annie Hsin-Wen Liu, Julian Bunn, and K. Mani Chandy. 2010. An Analysis of Data Fusion For Radiation Detection and Localization. In Proceedings 13th Conference on Information Fusion (FUSION'10). ISIF.
[17]
David C. Luckham. 2001. The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
[18]
George H. Mealy. 1955. A method for synthesizing sequential circuits. Bell System Technical Journal, The 34, 5 (Sept 1955), 1045--1079.
[19]
J. Misra and K. M. Chandy. 1981. Proofs of Networks of Processes. IEEE Trans. Softw. Eng. 7, 4 (July 1981), 417--426.
[20]
Shadi A. Noghabi, Kartik Paramasivam, Yi Pan, Navina Ramesh, Jon Bringhurst, Indranil Gupta, and Roy H. Campbell. 2017. Samza: Stateful Scalable Stream Processing at LinkedIn. Proc. VLDB Endow. 10, 12 (Aug. 2017), 1634--1645.
[21]
Michael Olson, Matthew Faulkner, Annie Hsin-Wen Liu, and K. Mani Chandy. {n. d.}. Handbook on Securing Cyber-Physical Infrastructure. Morgan Kaufman, Chapter 21.
[22]
Amir Pnueli. 1977. The Temporal Logic of Programs. In Proceedings of the 18th Annual Symposium on Foundations of Computer Science (SFCS '77). IEEE Computer Society, Washington, DC, USA, 46--57.
[23]
A. P. Sistla and E. M. Clarke. 1985. The Complexity of Propositional Linear Temporal Logics. J. ACM 32, 3 (July 1985), 733--749.
[24]
Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M. Patel, Sanjeev Kulkarni, Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, Nikunj Bhagat, Sailesh Mittal, and Dmitriy Ryaboy. 2014. Storm@Twitter. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SIGMOD '14). ACM, New York, NY, USA, 147--156.
[25]
Min-Hao Matt Wu, Annie Hsin-Wen Liu, and K. Mani Chandy. 2005. Virtual Environments for Developing Strategies for Interdicting Terrorists Carrying Dirty Bombs. In Proceedings of the 5th International Conference on Information Systems for Crisis Response and Management (ISCRAM'05). ISCRAM.
[26]
Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Computing with Working Sets. In Proceedings of the 2Nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud'10). USENIX Association, Berkeley, CA, USA, 10--10. http://dl.acm.org/citation.cfm?id=1863103.1863113
Recommendations
Scheduling intense applications most 'surprising' first
Certain streaming applications are required to perform sophisticated analytics within bounded time on arriving streams of data. Such applications have the interesting characteristic that the total amount of work that could be performed is unbounded. We ...
Scheduling intense applications most 'surprising' first
SAC '10: Proceedings of the 2010 ACM Symposium on Applied ComputingCertain streaming applications are required to perform sophisticated analytics within bounded time on arriving streams of data. Such applications have the interesting characteristic that the total amount of work that could be performed is unbounded. We ...
Substream management in distributed streaming dataflows
DEBS '22: Proceedings of the 16th ACM International Conference on Distributed and Event-Based SystemsMost state-of-the-art SPEs use punctuations to divide a stream into bounded substreams of messages, such as epochs and windows. The punctuation approach is powerful but has limitations: it does not support cyclic dataflows, is poorly scalable in some ...
Comments
Information & Contributors
Information
Published In
January 2019
535 pages
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
- SIGOPS: ACM Special Interest Group on Operating Systems
- Indian Institute of Science: Indian Institute of Science
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 January 2019
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICDCN '19
Sponsor:
- SIGOPS
- Indian Institute of Science
ICDCN '19: International Conference on Distributed Computing and Networking
January 4 - 7, 2019
Bangalore, India
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 90Total Downloads
- Downloads (Last 12 months)5
- Downloads (Last 6 weeks)0
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in