Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

JBCN barcelona 2017 kappa architecture 2.0

•
•
Juantomás García Molina
Juantomás García Molina

This document provides an overview of Kappa Architecture presented by Juantomás García. It includes: 1) A brief history of Kappa Architecture, which was coined in 2014 by Jay Kreps to describe an architecture using real-time streaming data and batch processing. 2) An explanation of how Kappa Architecture works, using streaming data pipelines to continuously update real-time views and batch jobs to rebuild views from historical data. 3) A real use case example of how OpenSistemas used Kappa Architecture to monitor vehicle data from many cars in real-time and perform analytics.

1 of 33
Download to read offline
Juantomás García - Open Sistemas Kappa Architecture 2.0 JBCN 2017 Barcelona
Hola Barcelona first
Juantomás García • Data Solutions Manager @ OpenSistemas • GDE (Google Developer Expert) for cloud Others • Co-Author of the first Spanish free software book “La Pastilla Roja” • President of Hispalinux (Spanish Linux User Group) • Organizer of the Machine Learning Spain and GDG Cloud Madrid. Who I am
• A brief history of Kappa Architecture • How we do Kappa Architecture • A little real example • Another ways to implement it. Agenda
What’s Kappa Architecture? July 2, 2014 Jay Kreps coined the term Kappa Architecture in an article for O’reilly Radar “Maybe we could call this the Kappa Achitecture, though it may be too simple of an idea to merit a Greek letter”
Jay has been involved in lots of projects: ✓ Author of the essay: The Log: What every software engineer should know about real-time data's unifying abstraction (12/16/2013) ✓ Author of the book I love Logs Who is Jay Kreps?
•Involved with projects as: ✓ Apache Kafka ✓ Apache Samza ✓ Voldemort ✓ Azkaban ✓ Ex-Linkedin ✓ Now co-founder and CEO of Confluent Who is Jay Kreps?
Usual Data Flow
Usual Data Flow
Usual Data Flow
Kappa Architecture Way
Tools we use
Tools we use
Tools we use
✓ If you have an schema spark SQL, is perfect. ✓ Spark streaming works very fine with spark and almost each streaming sources. ✓ Structured queries will be a huge advance. ✓ We love Scala, the spirit of Spark. Some Favorite Spark Features
We love code like this: Some Favorite Spark Features
• One of our clients wanted to monitor all the car's information via OBD II • OBD II is a car interface with the car electronics. • Our client developed an app for reading all the car information throw ODB II with bluetooth A Real Use Case
A Real Use Case
• We needed to scale the rest interfaces. There were too many requests. • MySQL don’t scale • Client wanted to do realtime expensive queries. First Problems
Some metrics
Architecture v 2.0
Architecture v 3.0
We can have queries like: “What are the drivers that are not client of the X gas brand, has a few gas and are near of gas station of the brand X and if true, send a notification with a discount coupon and a link with the route." Now we’re more flexible!!
• Kappa architecture is not a silver bullet but helps with a lot of solutions. • Kafka + spark streaming are our favorite tools • There are a lots of improvements: Takeaways ✓ OLAP like Apache Druid ✓ Graph databases like neo4j ✓ Kafka streams and compacts logs ✓ Apache Beams ✓ Scio Scala bindings
Takeaways: Apache Beam
Takeaways: Scio Scala Binding
Think Big
Think Big • Forget Legacy Architectures • Forget Old Tools • Use Light Technologies / Serverless • Use pieces of Lego • Mix different technologies from diverse sources
Spark Use Cases Not to do list •Avoid install & config a server even a VM. •Avoid installs tools instead use containers and/or cloud services. •In general: think if there is a simpler way to do it and needs less effort
Spark Use Cases Architecture & Tools •To use Cloud Services is not a brainer decision. •Git + Containers + Kubernetes •Use the best language* for each module. •Use Notebooks: Jupyter, Zeppelin, DSX (*) Even java might be an option - unprovable
Google Cloud Version
Kappa Architecture Questions? •email: juantomas@opensistemas.com •twitter: @juantomas This talk have a free questions lifetime warranty: If you have any questions or concerns about this talk, feel free to contact me anytime. Selfie Time: If you like the talk just smile while I take the selfie ;-)
Kappa Architecture Thank you

More Related Content

JBCN barcelona 2017 kappa architecture 2.0

  • 1. Juantomás García - Open Sistemas Kappa Architecture 2.0 JBCN 2017 Barcelona
  • 3. Juantomás García • Data Solutions Manager @ OpenSistemas • GDE (Google Developer Expert) for cloud Others • Co-Author of the first Spanish free software book “La Pastilla Roja” • President of Hispalinux (Spanish Linux User Group) • Organizer of the Machine Learning Spain and GDG Cloud Madrid. Who I am
  • 4. • A brief history of Kappa Architecture • How we do Kappa Architecture • A little real example • Another ways to implement it. Agenda
  • 5. What’s Kappa Architecture? July 2, 2014 Jay Kreps coined the term Kappa Architecture in an article for O’reilly Radar “Maybe we could call this the Kappa Achitecture, though it may be too simple of an idea to merit a Greek letter”
  • 6. Jay has been involved in lots of projects: ✓ Author of the essay: The Log: What every software engineer should know about real-time data's unifying abstraction (12/16/2013) ✓ Author of the book I love Logs Who is Jay Kreps?
  • 7. •Involved with projects as: ✓ Apache Kafka ✓ Apache Samza ✓ Voldemort ✓ Azkaban ✓ Ex-Linkedin ✓ Now co-founder and CEO of Confluent Who is Jay Kreps?
  • 15. ✓ If you have an schema spark SQL, is perfect. ✓ Spark streaming works very fine with spark and almost each streaming sources. ✓ Structured queries will be a huge advance. ✓ We love Scala, the spirit of Spark. Some Favorite Spark Features
  • 16. We love code like this: Some Favorite Spark Features
  • 17. • One of our clients wanted to monitor all the car's information via OBD II • OBD II is a car interface with the car electronics. • Our client developed an app for reading all the car information throw ODB II with bluetooth A Real Use Case
  • 18. A Real Use Case
  • 19. • We needed to scale the rest interfaces. There were too many requests. • MySQL don’t scale • Client wanted to do realtime expensive queries. First Problems
  • 23. We can have queries like: “What are the drivers that are not client of the X gas brand, has a few gas and are near of gas station of the brand X and if true, send a notification with a discount coupon and a link with the route." Now we’re more flexible!!
  • 24. • Kappa architecture is not a silver bullet but helps with a lot of solutions. • Kafka + spark streaming are our favorite tools • There are a lots of improvements: Takeaways ✓ OLAP like Apache Druid ✓ Graph databases like neo4j ✓ Kafka streams and compacts logs ✓ Apache Beams ✓ Scio Scala bindings
  • 28. Think Big • Forget Legacy Architectures • Forget Old Tools • Use Light Technologies / Serverless • Use pieces of Lego • Mix different technologies from diverse sources
  • 29. Spark Use Cases Not to do list •Avoid install & config a server even a VM. •Avoid installs tools instead use containers and/or cloud services. •In general: think if there is a simpler way to do it and needs less effort
  • 30. Spark Use Cases Architecture & Tools •To use Cloud Services is not a brainer decision. •Git + Containers + Kubernetes •Use the best language* for each module. •Use Notebooks: Jupyter, Zeppelin, DSX (*) Even java might be an option - unprovable
  • 32. Kappa Architecture Questions? •email: juantomas@opensistemas.com •twitter: @juantomas This talk have a free questions lifetime warranty: If you have any questions or concerns about this talk, feel free to contact me anytime. Selfie Time: If you like the talk just smile while I take the selfie ;-)