Streaming Processing in Uber Marketplace for Kafka Summit 2016
•
2 likes•629 views
This document discusses Uber's use of stream processing for their marketplace. It outlines several key use cases including real-time OLAP, complex event processing, and supply positioning. It then describes the challenges of processing large-scale geo-spatial temporal data in near real-time. The document proposes an overall architecture using Apache Kafka for event collection and Apache Samza for event processing. It notes some of the applications that can be built including dashboards, ad-hoc queries, and data visualizations. Finally, it discusses some trade-offs around using Lambda vs Kappa architectures and limitations of Samza.
Report
Share
Streaming Processing in Uber Marketplace for Kafka Summit 2016
- 2. ~ 68 countries / 350+ cities Transportation as reliable as running water, everywhere, for everyone 2
- 3. Agenda What’s on the menu? •Use Cases •Problem Space •Overall Architecture •Choices & Tradeoffs •Q & A
- 4. Use Case: Realtime OLAP
- 5. There is always need for quick exploration
- 6. How many open cars in the world, NOW?
- 8. How many UberXs were driving clients in SF in the past 10 minutes by hexagons?
- 9. How many UberXs were driving clients in SF in the past 10 minutes by hexagons?
- 10. Driving time and other metrics over time by hexagonal area
- 12. Use Case: Complex Event Processing
- 13. There are patterns in event streams
- 14. How many drivers cancel requests more than 3 times in a row within a 10- minute window?
- 15. Report riders requesting a pickup 100 miles apart within a half hour window?
- 16. IF This —> Then that —> ● Sigma is similar - but for offline/batch applications Complex Event Processing
- 17. Use Case: Supply Positioning
- 18. Clusters Of Supply & Demand
- 19. Predicted Health Metrics Actual Health Metrics Monitor Marketplace Health
- 20. Challenges
- 21. OLAP of Geo-spatial Temporal Data Reasonably Large Scale Near Real Time
- 22. • Indexing, Lookup, Rendering • Symmetric Neighbors • Convex & Compact Regions • Equal Areas • Equal Shape Hexagons
- 23. Scale Geo Space Vehicle Types Time Status X X X
- 25. Granular Geo Areas Over 10,000 hexagons in a city
- 26. Multiple Vehicle Types 7 vehicle types
- 27. Minute-level Time Buckets 1440 minutes in a day
- 28. Many Driver States 13 driver states
- 30. Granular Data 1 day of data: 300 x 10,000 x 7 x 1440 x 13 = 393 billion possible combinations
- 31. Unknown Query Patterns Any combination of dimensions
- 32. Variety of Aggregations - Heatmap - Top N - Histogram - count(), avg(), sum(), percent(), geo
- 33. Large Data Volume • Hundreds of thousands of events per second • At least dozens of fields in each event
- 34. Multiple Topics Rider States Driver States
- 35. Let’s build a stream processing pipeline
- 36. Accurate Statistics • E.g., can’t over count
- 38. Event Collection
- 39. Multiple Event Types with Different Volume
- 40. Hundreds of Thousands of Events Per Second
- 41. Events Should Be Available Under a Second
- 42. Events Should Rarely Get Lost
- 45. Natural Choice: Apache Kafka - Low latency and high throughput - Persistent events - Distributes a topic by partitions - Groups consumers by consumer groups
- 47. Event Processing
- 48. Transformation
- 49. Event Transformation Example (Lat, Long) -> (zipcode, hexagon, S2)
- 50. Pre-aggregation
- 52. Sessionization
- 54. Minimum Requirements - Statement Management - Checkpointing - Automatic Resource Management - Multi-staged processing
- 55. Apache Samza
- 56. Why Apache Samza? - DAG on Kafka - Excellent integration with Kafka - Built-in checkpointing - Built-in state management - Excellent support from our data team
- 57. Samza Is Conceptually Simple
- 58. IF This —> Then that —> ● Sigma is similar - but for offline/batch applications Complex Event Processing
- 59. ● Sigma is similar - but for offline/batch applications Complex Event Processing
- 60. ● Sigma is similar - but for offline/batch applications Complex Event Processing
- 61. ● Sigma is similar - but for offline/batch applications Complex Event Processing
- 62. ● Sigma is similar - but for offline/batch applications Complex Event Processing
- 63. ● Sigma is similar - but for offline/batch applications Slightly Expanded Version
- 64. ● Sigma is similar - but for offline/batch applications Slightly Expanded Version
- 65. ● Sigma is similar - but for offline/batch applications Slightly Expanded Version
- 66. ● Sigma is similar - but for offline/batch applications Slightly Expanded Version
- 68. Applications
- 69. Dashboard of Realtime Business Metrics
- 70. Ad-Hoc Queries
- 79. A Few Trade-Offs
- 80. Lambda vs Kappa
- 81. We Use Lambda - Spark + HDFS/S3 for batch processing - Yes, it is painful, but - We may need to go way back due to change of business requirements - Batch process can run faster — they scale differently - It was not easy to start a new stream processing instance
- 82. Processing by Event Time Is Not Always Easy
- 83. Leverage The Storage Layer
- 84. Dealing with Limitation of Samza -No broadcasting. We have to override SystemStreamPartitionGrouper -No dynamic topology. Can’t have arbitrary number of nested CEP queries -Tedious configuration and deployment of jobs. In house code-gem and deployment solution
- 85. Thank You