research-article

Real-time social network graph analysis using StreamMine3G

Authors:

Christof FetzerAuthors Info & Claims

DEBS '16: Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems

Pages 322 - 329

https://doi.org/10.1145/2933267.2933514

Published: 13 June 2016 Publication History

Abstract

In this paper, we present our approach for solving the DEBS Grand Challenge 2016 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant event stream processing (ESP) system. We first provide an overview about StreamMine3G with regards to its programming model and architecture, followed by thorough description of the implementation for the two queries that provide up-to-date information about (i) the top-3 active posts and (ii) the top-k comments with the largest maximum cliques. Novel aspects of our implementation include (i) highly optimized data structures that lower the amount of lookups and traversals, and a (ii) deterministic data partitioning and processing scheme that allows the system to scale without bounds in an elastic fashion while still guaranteeing semantic transparency. In order to better utilize nowadays many-core machines, we furthermore propose a pipelining scheme in addition to data partitioning. Finally, we present a brief performance evaluation of our system.

References

[1]

Apache flink: Scalable batch and stream data processing. https://flink.apache.org/, 2016.

[2]

Apache s4 - distributed stream computing platform. https://incubator.apache.org/s4/, 2016.

[3]

Apache samza - a distributed stream processing framework. http://samza.incubator.apache.org/, 2016.

[4]

Apache storm - distributed and fault-tolerant realtime computation. https://storm.incubator.apache.org/, 2016.

[5]

Redis. http://redis.io/, 2016.

[6]

Spark streaming. http://spark.apache.org/streaming/, 2016.

[7]

C. Bron and J. Kerbosch. Algorithm 457: Finding all cliques of an undirected graph. Commun. ACM, 16(9):575--577, Sept. 1973.

Digital Library

[8]

J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Commun. ACM, 51(1):107--113, Jan. 2008.

Digital Library

[9]

A. Galeos, P. Gryllos, N. Leventis, K. Mavrikis, and S. Voulgaris. Pimp my taxi ride. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS '15, pages 318--319, New York, NY, USA, 2015. ACM.

Digital Library

[10]

V. Gulisano, Z. Jerzak, S. Voulgaris, and H. Ziekow. The debs grand challenge 2016. In Proceedings of the 10th ACM International Conference on Distributed Event-Based Systems, DEBS '16, New York, NY, USA, 2016. ACM.

Digital Library

[11]

J. Konc and D. Janezic. An improved branch and bound algorithm for the maximum clique problem. MATCH Communications in Mathematical and in Computer Chemistry, June 2007.

[12]

A. Martin, A. Brito, and C. Fetzer. Real time data analysis of taxi rides using streammine3g. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS '15, pages 269--276, New York, NY, USA, 2015. ACM.

Digital Library

[13]

L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream computing platform. In Proceedings of the 2010 IEEE International Conference on Data Mining Workshops, ICDMW '10, pages 170--177, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

[14]

A. Suriarachchi and S. Pallickara. A high-throughput, scalable solution for calculating frequent routes and profitability of new york taxis. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS '15, pages 301--308, New York, NY, USA, 2015. ACM.

Digital Library

[15]

M. Thompson, D. Farley, M. Barker, P. Gee, and A. Stewart. DISRUPTOR: High performance alternative to bounded queues for exchanging data between concurrent threads.

Cited By

Ehmes SFritsche LAltenhofen K(2020)GrapeL: Combining Graph Pattern Matching and Complex Event ProcessingSystems Modelling and Management10.1007/978-3-030-58167-1_13(180-196)Online publication date: 17-Oct-2020
https://doi.org/10.1007/978-3-030-58167-1_13
Marozzo FBelcastro L(2019)Cloud Computing for Big Data AnalysisEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_136(452-457)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_136
Marozzo FBelcastro L(2018)Cloud Computing for Big Data AnalysisEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_136-1(1-6)Online publication date: 14-Feb-2018
https://doi.org/10.1007/978-3-319-63962-8_136-1
Show More Cited By

Index Terms

Real-time social network graph analysis using StreamMine3G
1. Information systems
  1. Information systems applications
    1. Spatial-temporal systems
      1. Data streaming
2. Theory of computation
  1. Design and analysis of algorithms
    1. Distributed algorithms

Recommendations

Real time data analysis of taxi rides using StreamMine3G
DEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

In this paper, we present our approach for solving the DEBS Grand Challenge 2015 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We first provide an overview about the system architecture of StreamMine3G ...
Predicting energy consumption with StreamMine3G
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

In this paper, we present our approach on solving the DEBS Grand Challenge using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We will provide an overview about the system architecture of Stream-Mine3G and ...
Scalable and elastic realtime click stream analysis using StreamMine3G
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

Click stream analysis is a common approach for analyzing customer behavior during the navigation through e-commerce or social network sites. Performing such an analysis in real-time opens up new business opportunities as well as increases revenues as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DEBS '16: Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems

June 2016

456 pages

ISBN:9781450340212

DOI:10.1145/2933267

General Chairs:
Avigdor Gal
Technion, Israel
,
Matthias Weidlich
Humboldt-Universität zu Berlin, Germany
,
Program Chairs:
Vana Kalogeraki
Athens University of Economics and Business, Greece
,
Nalini Venkasubramanian
University of California, Irvine

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Community
German Excellence Initiative, Center for Advancing Electronics Dresden (cfAED)

Conference

DEBS '16

Sponsor:

DEBS '16: The 10th ACM International Conference on Distributed and Event-based Systems

June 20 - 24, 2016

California, Irvine

Acceptance Rates

Overall Acceptance Rate 145 of 583 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
151
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ehmes SFritsche LAltenhofen K(2020)GrapeL: Combining Graph Pattern Matching and Complex Event ProcessingSystems Modelling and Management10.1007/978-3-030-58167-1_13(180-196)Online publication date: 17-Oct-2020
https://doi.org/10.1007/978-3-030-58167-1_13
Marozzo FBelcastro L(2019)Cloud Computing for Big Data AnalysisEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_136(452-457)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_136
Marozzo FBelcastro L(2018)Cloud Computing for Big Data AnalysisEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_136-1(1-6)Online publication date: 14-Feb-2018
https://doi.org/10.1007/978-3-319-63962-8_136-1
Belcastro LMarozzo FTalia DTrunfio P(2017)Big Data Analysis on CloudsHandbook of Big Data Technologies10.1007/978-3-319-49340-4_4(101-142)Online publication date: 26-Feb-2017
https://doi.org/10.1007/978-3-319-49340-4_4
Marozzo FBelcastro L(2012)Cloud Computing for Big Data AnalysisEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_136-2(1-7)Online publication date: 24-Feb-2012
https://doi.org/10.1007/978-3-319-63962-8_136-2

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents