DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García

•

3 likes•498 views

DataScience Lab, 13 мая 2017 Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García (Data Solutions Manager at OpenSistemas, Madrid, Spain) We will have an introduction of what is the kappa architecture vs lambda architecture. We will see how kappa architecture is a good solution to implement solutions in (almost) real time when we need to analyze data in streaming. We will show in a case of real use: how architecture is designed, how pipelines are organized and how data scientists use it. We will review the most used technologies to implement it from apache Kafka + spark using Scala to new tools like apache beam / google dataflow. Все материалы: http://datascience.in.ua/report2017

Juantomás García - Open Sistemas
Kappa Architecture 2.0
DataScience Lab, Odessa

Доброго ранку
Одеса!!
(Dobroho ranku Odesa)
first

Juantomás García
• Data Solutions Manager @ OpenSistemas
• GDE (Google Developer Expert) for cloud
Others
• Co-Author of the first Spanish free software book “La Pastilla
Roja”
• President of Hispalinux (Spanish Linux User Group)
• Organizer of the Machine Learning Spain and GDG Cloud
Madrid.
Who I am

What’s Kappa Architecture?
July 2, 2014 Jay Kreps coined the term Kappa
Architecture in an article for O’reilly Radar
“Maybe we could call this the Kappa Achitecture, though it may
be too simple of an idea to merit a Greek letter”

Jay has been involved in lots
of projects:
✓ Author of the essay: The
Log: What every software
engineer should know about
real-time data's unifying
abstraction (12/16/2013)
✓ Author of the book I love
Logs
Who is Jay Kreps?

•Involved with projects as:
✓ Apache Kafka
✓ Apache Samza
✓ Voldemort
✓ Azkaban
✓ Ex-Linkedin
✓ Now co-founder and CEO of Confluent
Who is Jay Kreps?

✓ If you have an schema spark SQL, is
perfect.
✓ Spark streaming works very fine with spark
and almost each streaming sources.
✓ Structured queries will be a huge advance.
✓ We love Scala, the spirit of Spark.
Some Favorite Spark Features

We love code like this:
Some Favorite Spark Features

• One of our clients wanted to monitor all the
car's information via OBD II
• OBD II is a car interface with the car
electronics.
• Our client developed an app for reading all
the car information throw ODB II with
bluetooth
A Real Use Case

• We needed to scale the rest interfaces.
There were too many requests.
• MySQL don’t scale
• Client wanted to do realtime expensive
queries.
First Problems

• Kappa architecture is not a silver bullet but helps
with a lot of solutions.
• Kafka + spark streaming are our favorite tools
• There are a lots of improvements:
Takeaways
✓ OLAP like Apache Druid
✓ Graph databases like neo4j
✓ Kafka streams and
compacts logs
✓ Apache Beams
✓ Scio Scala bindings

Think Big
• Forget Legacy Architectures
• Forget Old Tools
• Use Light Technologies / Serverless
• Use pieces of Lego
• Mix different technologies from diverse sources

Spark Use Cases
Not to do list
•Avoid install & config a server even a
VM.
•Avoid installs tools instead use
containers and/or cloud services.
•In general: think if there is a simpler
way to do it and needs less effort

Spark Use Cases
Architecture & Tools
•To use Cloud Services is not a brainer
decision.
•Git + Containers + Kubernetes
•Use the best language* for each
module.
•Use Notebooks: Jupyter, Zeppelin,
DSX
(*) Even java might be an option - unprovable

Kappa Architecture
Questions?
•email: juantomas@opensistemas.com
•twitter: @juantomas
This talk have a free questions lifetime warranty: If you have any questions or concerns
about this talk, feel free to contact me anytime.
Selfie Time: If you like the talk just smile while I take
the selfie ;-)

What's hot

Augmenting Mongo DB with treasure data

Treasure Data, Inc.

Large Scale Graph Analytics with JanusGraph

P. Taylor Goetz

This document provides an overview of large scale graph analytics and JanusGraph. It discusses graph databases and their use cases. JanusGraph is presented as an open source graph database that can scale to billions of vertices and edges across multiple storage backends like HBase, Cassandra and Bigtable. It uses the TinkerPop framework and Gremlin query language. JanusGraph supports ACID transactions, external indices, and evolving schemas. Example graph queries are demonstrated using the Gremlin console.

Neo4j tms

_mdev_

A (XPages) developers guide to Cloudant

Frank van der Linden

Frank van der Linden presented on connecting XPages applications to Cloudant. He began with an introduction to Cloudant, describing it as the cloud version of CouchDB that stores data as JSON documents. He then covered how to connect to Cloudant directly via REST or through an OSGi plugin, and described storing and retrieving data from Cloudant using a Java connector. Finally, he demonstrated integrating Cloudant with an XPages application to store and search job documents, attachments, and rich text.

Unifying Events and Logs into the Cloud

Eduardo Silva Pereira

1. Eduardo Silva discussed unifying event and log data from multiple sources into the cloud using Fluentd and Fluent Bit. 2. Fluentd is an open source data collector that allows for parsing and storing data from multiple sources through its pluggable input and output plugins. 3. Fluent Bit is designed for collecting data from IoT and embedded devices to transport it to third party services, with a focus on performance and lightweight resource usage.

Briney - Leveling Up Data Management - With Notes

National Information Standards Organization (NISO)

Elk meetup

Asaf Yigal

Serverless Big Data Architecture on Google Cloud Platform at Credit OK

Kriangkrai Chaonithi

Serverless Big Data Architecture on Google Cloud Platform was presented by Kriangkrai Chaonithi. The presentation covered Credit OK's use of serverless architecture on GCP for their big data analytics platform. Credit OK processes large amounts of customer data from over 400 sites to perform credit scoring. They use Google Cloud Functions to ingest data from sites, as well as Compute Engine and Google Cloud Storage. This serverless architecture allows them to automatically scale infrastructure as needed, reducing costs since they only pay for resources used. While serverless architectures don't require managing servers, there are still resource limits that must be considered to avoid issues like exhausted worker pools during peak loads.

A (XPages) developers guide to Cloudant - MeetIT

Frank van der Linden

This document provides an overview of using Cloudant as a NoSQL database with XPages applications. It discusses why the author chose Cloudant for an HR application, introduces Cloudant and how it works, explains how to connect to Cloudant from XPages using a Cloudant connector plugin, and demonstrates storing and retrieving jobs, attachments, and search data from Cloudant in an XPages application. The document contains code examples for Cloudant DAOs, controllers, and services to interact with Cloudant from the backend and display data in XPages.

Protecting Your Cluster from Your Humans

Elasticsearch

Indexing big data in the cloud

OpenSource Connections

Amazon Web Services offers a quick and easy way to build a scalable search platform, a flexibility is especially useful when an initial data load is required but the hardware is no longer needed for day-to-day searching and adding new documents. This presentation will cover one such approach capable of enlisting hundreds of worker nodes to ingest data, track their progress, and relinquish them back to the cloud when the job is done. The data set that will be discussed is the collection of published patent grants available through Google Patents. A single Solr instance can easily handle searching the roughly 1 million patents issued between 2010 and 2005, but up to 50 worker nodes were necessary to load that data in a reasonable amount of time. Also, the same basic approach was used to make three sizes of PNG thumbnails of the patent grant TIFF images. In that case 150 worker nodes were used to generate 1.6 Tb of data over the course of three days. In this session, attendees will learn how to leverage EC2 as a scalable indexer and tricks for using XSLT on very large XML documents.

2018 01-15 infra coders meetup vienna ii

Aleksandar Lazic

This document provides an overview and comparison of the Gluster and Ceph distributed storage systems. It discusses the history and architecture of each system. Both Gluster and Ceph were created in the 2000s as software-defined storage solutions and later acquired by Red Hat. Key differences are that Gluster provides a POSIX filesystem and FUSE mount while Ceph is an object store with a REST interface. Both can be used natively with Kubernetes and OpenShift. The document concludes with information on demoing Gluster storage on Oracle Cloud Infrastructure.

DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...

Frank van der Linden

The role of a human resources employee can be hard work when it comes to filtering hundreds or thousands of job applications. Often, the cover letters are submitted in unstructured formats, making data organization and identification of interesting job applications difficult. In this session, we will share a look behind the scenes of the award-winning HR Assistant application, which uses a combination of IBM Bluemix, Watson and a Cloudant database to improve the recruitment process. See how we integrate these different technologies and display the content graphically using XPages, along with how the development progressed and the challenges we faced.

Find your data

Oliver Busse

This document summarizes Oliver Busse's presentation on using Graph databases in XPages applications. It introduces Graph databases and their terminology like vertices and edges. It describes how GraphNSF, a Graph database implementation in Domino, stores vertices and edges as documents in NSF files. It provides examples of creating vertices and edges programmatically and adding metadata to existing data using a graph structure. It concludes with a live demo of a GraphNSF implementation in an XPages application.

Atlogys Academy - Tech Talk on Mongo DB

Atlogys Technical Consulting

Shivam Singhal - Senior Engineer at Atlogys delivers a highly informative, practical and hands-on workshop cum talk on Mongo DB and how it can be & should be used in making real world web applications. Part of the Atlogys Academy series, this talk covers the fundamentals of this noSQL technology and then goes on to show many practical examples on the command line as well. A must watch for anyone working with Mongo DB.

Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

mortardata

Presentation from AWS re: Invent, where Mortar announced it's public availability on stage. Mortar facilitates Hadoop data development in two parts: an open source data development framework (like Rails for Hadoop) and a PaaS (like Heroku for Hadoop). The open source framework allows for sharing/repeating/maintaining code history, automated testing, and one-button deploy. The PaaS provides high availability execution, full job history, and helps users identify problems.

JanusGraph, Jupyter Meetup NYC

Jason Plurad

Going Serverless with AWS Lambda at ReportGarden

Jay Gandhi

This document discusses using AWS Lambda for serverless web scraping. It provides an example use case of scraping business directory websites to check for incorrect or missing business listings. Traditionally this would require writing separate code for each directory, but with serverless it can be done more flexibly. The benefits of the serverless approach include increased scalability, reduced costs since functions only run when needed, and easier integration with other AWS services. The document demonstrates setting up a serverless framework for asynchronous web scraping using AWS Lambda, SNS, and API Gateway.

EDU2.0 and Amazon CloudSearch

Michael Bohlig

This document discusses migrating an e-learning platform's search from Sphinx to Amazon CloudSearch. The platform has over 1 million users and indexes data from MySQL and DynamoDB. The migration involved testing with one item type, using a script to upload initial contents, then indexing all other types except high-volume messages and postings which were migrated last. Ruby on Rails was integrated using a gem and background workers update the search every 15 minutes. The migration took two weeks and simplified the architecture for better scalability.

What's hot (19)

Augmenting Mongo DB with treasure data

Large Scale Graph Analytics with JanusGraph

Neo4j tms

A (XPages) developers guide to Cloudant

Unifying Events and Logs into the Cloud

Briney - Leveling Up Data Management - With Notes

Elk meetup

Serverless Big Data Architecture on Google Cloud Platform at Credit OK

A (XPages) developers guide to Cloudant - MeetIT

Protecting Your Cluster from Your Humans

Indexing big data in the cloud

2018 01-15 infra coders meetup vienna ii

DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...

Find your data

Atlogys Academy - Tech Talk on Mongo DB

Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

JanusGraph, Jupyter Meetup NYC

Going Serverless with AWS Lambda at ReportGarden

EDU2.0 and Amazon CloudSearch

Similar to DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García

Kappa Architecture, IoT of the cars - LibreCon 2016

LibreCon

This document discusses Kappa Architecture, which is a data processing architecture using Apache Kafka as the centralized data backbone. It summarizes Jay Kreps' original definition of Kappa Architecture and describes how it provides a more flexible way to do real-time and batch processing compared to the traditional Lambda Architecture. The document also provides an example of how a company implemented Kappa Architecture using Kafka and Spark Streaming to monitor car telemetry data from many vehicles and support flexible real-time queries over that data.

Architecting Your First Big Data Implementation

Adaryl "Bob" Wakefield, MBA

This document provides an overview of architecting a first big data implementation. It defines key concepts like Hadoop, NoSQL databases, and real-time processing. It recommends asking questions about data, technology stack, and skills before starting a project. Distributed file systems, batch tools, and streaming systems like Kafka are important technologies for big data architectures. The document emphasizes moving from batch to real-time processing as a major opportunity.

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Sarah Guido

Given at Data Day Seattle 2015. Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.

Cloud Big Data Architectures

Lynn Langit

This document provides an overview of a workshop on cloud big data architectures. The workshop covers: 1. Different types of big data solutions and when to use each, such as Hadoop, NoSQL and big relational databases. 2. Data pipelines, including ETL tools, load testing patterns and connecting clouds. 3. Querying and visualizing data through business analytics, predictive analytics and visualization tools. 4. A brief introduction to IoT and how it relates to big data.

Beyond Relational

Lynn Langit

The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.

ASPgems - kappa architecture

Juantomás García Molina

Kappa Architecture is an alternative to Lambda Architecture that simplifies real-time data processing. It uses a distributed log like Kafka to store all input data immutably to allow reprocessing from the beginning if the processing code changes. This avoids having to maintain separate batch and real-time processing systems. The ASPgems team has implemented Kappa Architecture for several clients using Kafka, Spark Streaming, and Cassandra to provide real-time analytics and metrics in sectors like telecommunications, IoT, insurance, and energy.

Apache spark y cómo lo usamos en nuestros proyectos

OpenSistemas

This document discusses Apache Spark and how it is used in projects at OpenSistemas. It provides a brief history of Hadoop and the limitations that led to the development of Apache Spark. It then explains what Apache Spark is, how it was designed from the ground up to take advantage of memory, and how it is influenced by Scala. It also introduces the Kappa architecture as an alternative to the Lambda architecture that only requires maintaining a single stream processing codebase. Finally, it provides examples of how OpenSistemas uses the Kappa architecture with Apache Spark streaming for various clients in domains like telecommunications, insurance, energy, and more.

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...

Radhika Puthiyetath

Sebastien Goasguen is a developer who works on Apache CloudStack and other open source projects related to cloud computing, containers, and big data. He gave a talk covering CloudStack, Docker, Kubernetes, CoreOS, and how various technologies can work together for managing distributed applications and infrastructure. He also discussed the evolving landscape of cloud computing and how big data solutions fit within that landscape.

Which database should I use for my app?

Nawaz Dhandala

Cloud and Big Data trends

Sebastien Goasguen

A look at clouds and big data trends and history. While Big Data arrived first on the scene -looking at google file system, hadoop, dynamo- Cloud was first in the hyper cycle. Google trends show this clearly. Amazon AWS however has already deployed analytics services on the their cloud while open source IaaS solutions are still struggling to deliver a EC2 clone. Cloud and Big data has three common points: 1-use an EC2 clone and a S3 clone (riakCS, glusterfs etc) to build a cloud 2-Use a big data solutions as a backend to your cloud to provide EBS or large scale image catalogue 3-deploy big data solutions on your cloud with tools like apache whirr, pallet, and newer devops tool chains with vagrant and co.

Mapping Life Science Informatics to the Cloud

Chris Dagdigian

This document discusses strategies for mapping informatics to the cloud. It provides 9 tips for doing so effectively. Tip 1 advises that high-performance computing and clouds require a new model where resources are dedicated to each application. Tip 2 recommends hybrid cloud approaches but cautions they are less usable than claimed and practical only sometimes. The document emphasizes the need to handle legacy codes in addition to new "big data" approaches.

Why Organizations are Looking at Alternative Database Technologies – Introduc...

DATAVERSITY

Decoupling Drupal - Drupal Camp Toronto 2014

Alex De Winne

Technologies for Data Analytics Platform

N Masahiro

This document discusses building a data analytics platform and summarizes various technologies that can be used. It begins by outlining reasons for analyzing data like reporting, monitoring, and exploratory analysis. It then discusses using relational databases, parallel databases, Hadoop, and columnar storage to store and process large volumes of data. Streaming technologies like Storm, Kafka, and services like Redshift, BigQuery, and Treasure Data are also summarized as options for a complete analytics platform.

Getting Started with Big Data in the Cloud

RightScale

Traveloka's data journey — Traveloka data meetup #2

Traveloka

Discover the journey that Traveloka's Data Team have taken so far and learn from our struggles and triumphs in managing Traveloka's burgeoning data! In this slide, you will learn more about the stories and lessons learned on building a scalable data pipeline at Traveloka. Presenters of the slide: Nisrina Luthfiyati - Data Engineer Rendy B. Junior - Data System Architect Wilson lauw - Lead Data Engineer To follow our LinkedIn page, visit bit.ly/TravelokaLinkedInPage Safe Harbor Statement Our discussion may include predictions, estimates or other information that might be considered conclusive. While these conclusive statements represent our current judgment on the best practices, they are subject to risks and uncertainties that could cause actual results to differ materially. You are cautioned not to place undue reliance on our statements, which reflect our opinions only as of the date of this presentation. Please keep in mind that we are not obligating ourselves to revise or publicly release the results of any revision to these presentation materials in light of new information or future events.

An Incomplete Data Tools Landscape for Hackers in 2015

Wes McKinney

Wes McKinney gives an overview of the current data analysis tools landscape in Python and R. He discusses essential Python packages like NumPy, pandas, and scikit-learn. For R, he covers packages in the "Hadley stack" like dplyr and ggplot2. IPython/Jupyter notebooks are also mentioned as a platform for interactive data analysis across languages. The talk aims to highlight trends, opportunities, and challenges in the open source data science tool ecosystem.

Comment choisir entre Parse, Heroku et AWS ?

TheFamily

Server’s variations bsw2015

Laurent Cerveau

- The document discusses different approaches to server infrastructure - the mobile/product-focused Parse approach, the Heroku "server is software" approach, and the AWS "climb the server mountain" approach. - With Parse, the focus is on providing backend services for mobile apps through an easy-to-use platform. Heroku emphasizes treating the server as software to be developed. AWS puts the user directly in contact with infrastructure components. - Each approach has tradeoffs in terms of learning curve, costs, flexibility, and how long they are suitable before needing to transition to another platform to handle more complex needs. Managing servers requires developing new skills around concepts like storage, security, deployment and more.

Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...

Edward Wilde

This talk introduces OpenFaaS, an open source platform for serverless functions. We begin by installing, running the platform and invoking a machine learning function within 60 second (ish). Next we step up and look at the high level platform architecture and key pillars of OpenFaaS: developer first, operator friendly and community focused. We close by looking at a multiuser platform built on top OpenFaas. The objective of this talk is to introduce you to OpenFaaS the serverless platform and OpenFaaS cloud the multi-user environment for practicing git ops with functions.

Similar to DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García (20)

Kappa Architecture, IoT of the cars - LibreCon 2016

Architecting Your First Big Data Implementation

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Cloud Big Data Architectures

Beyond Relational

ASPgems - kappa architecture

Apache spark y cómo lo usamos en nuestros proyectos

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...

Which database should I use for my app?

Cloud and Big Data trends

Mapping Life Science Informatics to the Cloud

Why Organizations are Looking at Alternative Database Technologies – Introduc...

Decoupling Drupal - Drupal Camp Toronto 2014

Technologies for Data Analytics Platform

Getting Started with Big Data in the Cloud

Traveloka's data journey — Traveloka data meetup #2

An Incomplete Data Tools Landscape for Hackers in 2015

Comment choisir entre Parse, Heroku et AWS ?

Server’s variations bsw2015

Docker and serverless Randstad Jan 2019: OpenFaaS Serverless: when functions ...

More from GeeksLab Odessa

DataScience Lab2017_Коррекция геометрических искажений оптических спутниковых...

DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García

More Related Content

What's hot

What's hot (19)

Similar to DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García

Similar to DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García (20)

More from GeeksLab Odessa

More from GeeksLab Odessa (20)

Recently uploaded

Recently uploaded (20)

DataScience Lab 2017_Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García