"$10 thousand per minute of downtime: architecture, queues, streaming and fintech", Max Baginskiy
•
0 likes•103 views
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless. As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency. We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Recommended
More Related Content
Similar to "$10 thousand per minute of downtime: architecture, queues, streaming and fintech", Max Baginskiy
Similar to "$10 thousand per minute of downtime: architecture, queues, streaming and fintech", Max Baginskiy (20)
More from Fwdays
More from Fwdays (20)
Recently uploaded
Recently uploaded (20)
"$10 thousand per minute of downtime: architecture, queues, streaming and fintech", Max Baginskiy
- 1. $10 thousand per minute of downtime: architecture, queues, streaming and fintech Max Baginskiy Solidgate
- 2. About me Head of Engineering PreviouslyTech Lead and Platform engineer Head of Engineering PreviouslyTech Lead and Platform engineer Head of Engineering PreviouslyTech Lead and Platform engineer 10 yrs in Software Engineering Last 6 years Go, fan of DevOps 10 yrs in Software Engineering Last 6 years Go, fan of DevOps 10 yrs in Software Engineering Last 6 years Go, fan of DevOps Build teams (5 teams, 30+ people hired) And architecture Build teams (5 teams, 30+ people hired) And architecture Build teams (5 teams, 30+ people hired) And architecture
- 3. Agenda Company intro Architecture of the system Queues and Streams to choose from Low latency streaming using outbox CDC to our solution - comparison Questions. Company intro Architecture of the system Queues and Streams to choose from Low latency streaming using outbox CDC to our solution - comparison Questions. Company intro Architecture of the system Queues and Streams to choose from Low latency streaming using outbox CDC to our solution - comparison Questions.
- 4. About company 7+ years online 7+ years online 7+ years online 70 engineers 50 SW engineers 20 Infra + Data engineers + AQA 70 engineers 50 SW engineers 20 Infra + Data engineers + AQA 70 engineers 50 SW engineers 20 Infra + Data engineers + AQA PCI DSS Compliant PCI DSS Compliant PCI DSS Compliant European Acquirer European Acquirer European Acquirer
- 6. Business figures 2.5b$ annually 2.5b$ annually 2.5b$ annually 15-18m tx monthly 15-18m tx monthly 15-18m tx monthly 10k$ 1 min of downtime 10k$ 1 min of downtime 10k$ 1 min of downtime 40+ integrated payment methods and providers 40+ integrated payment methods and providers 40+ integrated payment methods and providers
- 7. ALBTraffic We have 100x less traffic on ALB during high season than Shopify We have 100x less traffic on ALB during high season than Shopify We have 100x less traffic on ALB during high season than Shopify Stripe served 250mil API calls in 2020 perday Stripe served 250mil API calls in 2020 perday Stripe served 250mil API calls in 2020 perday
- 8. Kafka Producer We started integrating kafka lastyear We started integrating kafka lastyear We started integrating kafka lastyear 20 rps average 20 rps average 20 rps average 2 mil events perday 2 mil events perday 2 mil events perday
- 9. RabbitMQ Producer 100-120 rps average 100-120 rps average 100-120 rps average 10 mil events perday 10 mil events perday 10 mil events perday
- 10. Logs 1.5-2k rps of logs 1.5-2k rps of logs 1.5-2k rps of logs 150 mil events per day. 200-300 GB of logs daily 150 mil events per day. 200-300 GB of logs daily 150 mil events per day. 200-300 GB of logs daily
- 11. Architecture
- 13. Project “Taxer”
- 14. Non functional requirements Durability out of the box Durability out of the box Durability out of the box Queue replay Queue replay Queue replay Single active consumer support Single active consumer support Single active consumer support Easy to setup and to maintain Easy to setup and to maintain Easy to setup and to maintain Partitioning Partitioning Partitioning Easy scaling for publisher and consumer Easy scaling for publisher and consumer Easy scaling for publisher and consumer Extensiblity: schema registry support, dynamic routing, enrichment Extensiblity: schema registry support, dynamic routing, enrichment Extensiblity: schema registry support, dynamic routing, enrichment
- 15. NFR - explanation What if message is lost in between services while processing andwe retry payment? What if message is lost in between services while processing andwe retry payment? What if message is lost in between services while processing andwe retry payment? What if message is lost in between callback service and callback processor? What if message is lost in between callback service and callback processor? What if message is lost in between callback service and callback processor? What if message is lost in between payment and finance systems? What if message is lost in between payment and finance systems? What if message is lost in between payment and finance systems? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …? what if …what if …what if …?
- 16. RabbitMQ dive in Erlang Written in Erlang. Erlang made by Ericssonwhich makes telecommunication devices. Erlang Written in Erlang. Erlang made by Ericssonwhich makes telecommunication devices. Erlang Written in Erlang. Erlang made by Ericssonwhich makes telecommunication devices. Proof of fail-safety ATM AXD301 example, Calculated uptime 99,9999999%, only one problem permany years. Proof of fail-safety ATM AXD301 example, Calculated uptime 99,9999999%, only one problem permany years. Proof of fail-safety ATM AXD301 example, Calculated uptime 99,9999999%, only one problem permany years. Mnesia as storage Mnesia doesn’t support recovery from split brain and othertypes of failures. Mnesia as storage Mnesia doesn’t support recovery from split brain and othertypes of failures. Mnesia as storage Mnesia doesn’t support recovery from split brain and othertypes of failures.
- 17. RabbitMQ Durability Mechanisms Publisher confirms is a MUST have RabbitMQ can store data to Disk and different autoheal modes Different types of queues: Quorum, Mirrored Have Streaming in “beta”. Mechanisms Publisher confirms is a MUST have RabbitMQ can store data to Disk and different autoheal modes Different types of queues: Quorum, Mirrored Have Streaming in “beta”. Mechanisms Publisher confirms is a MUST have RabbitMQ can store data to Disk and different autoheal modes Different types of queues: Quorum, Mirrored Have Streaming in “beta”. What if publisher confirms disabled? Delivery after exchange might not happen Persistence might not happen Few replicas might not acknowledge message in Quorum Overwhelmed Clusterwill not accept messages but publisherwill not know. What if publisher confirms disabled? Delivery after exchange might not happen Persistence might not happen Few replicas might not acknowledge message in Quorum Overwhelmed Clusterwill not accept messages but publisherwill not know. What if publisher confirms disabled? Delivery after exchange might not happen Persistence might not happen Few replicas might not acknowledge message in Quorum Overwhelmed Clusterwill not accept messages but publisherwill not know.
- 18. Quorum queues + Pros Have Consensus built in Data written to disk, metadata in memory Can easily handle restarts. + Pros Have Consensus built in Data written to disk, metadata in memory Can easily handle restarts. + Pros Have Consensus built in Data written to disk, metadata in memory Can easily handle restarts. − Cons Doesn’t scale well - millions of messages after restart can replicate hours. Doen’t have “replay” mechanism Consumers doesn’t scale Doesn’t preserve order of messages. − Cons Doesn’t scale well - millions of messages after restart can replicate hours. Doen’t have “replay” mechanism Consumers doesn’t scale Doesn’t preserve order of messages. − Cons Doesn’t scale well - millions of messages after restart can replicate hours. Doen’t have “replay” mechanism Consumers doesn’t scale Doesn’t preserve order of messages.
- 19. Split brain
- 20. Split brain - autoheal ignore Usewhen network reliability is the highest practically possible and node availability is of topmost importance. ignore Usewhen network reliability is the highest practically possible and node availability is of topmost importance. ignore Usewhen network reliability is the highest practically possible and node availability is of topmost importance. pause_minority Appropriatewhen clustering across racks oravailability zones in a single region and the probability of losing a majority of nodes (zones) at once is considered to bevery low. pause_minority Appropriatewhen clustering across racks oravailability zones in a single region and the probability of losing a majority of nodes (zones) at once is considered to bevery low. pause_minority Appropriatewhen clustering across racks oravailability zones in a single region and the probability of losing a majority of nodes (zones) at once is considered to bevery low. autoheal Appropriatewhen are more concernedwith continuity of service thanwith data consistency across nodes. autoheal Appropriatewhen are more concernedwith continuity of service thanwith data consistency across nodes. autoheal Appropriatewhen are more concernedwith continuity of service thanwith data consistency across nodes. Summary - noway to guarantee that autohealwillwork properly Summary - noway to guarantee that autohealwillwork properly Summary - noway to guarantee that autohealwillwork properly
- 21. RabbitMQ streaming, problem #1
- 22. RabbitMQ streaming. Go client, issue #2
- 23. RabbitMQ streaming. Go client, issue #3
- 24. RabbitMQ + RabbitMQ streaming Newfeature that not a lot of companies use. Newfeature that not a lot of companies use. Newfeature that not a lot of companies use. Go client is not ready,what about Python orNode.js I’m aftaid to ask. Go client is not ready,what about Python orNode.js I’m aftaid to ask. Go client is not ready,what about Python orNode.js I’m aftaid to ask. Hard to support. Requires updates of Erlang and then RabbitMQ. Hard to support. Requires updates of Erlang and then RabbitMQ. Hard to support. Requires updates of Erlang and then RabbitMQ. Streaming is a plugin that requires specificversion of RabbitMQ. Streaming is a plugin that requires specificversion of RabbitMQ. Streaming is a plugin that requires specificversion of RabbitMQ. Not made for fintech: lack of properdurability, lack of functionality. Not made for fintech: lack of properdurability, lack of functionality. Not made for fintech: lack of properdurability, lack of functionality.
- 26. Kafka dive in Java Written in Java by Linkedin and then opensourced and licenced under Apache licence. Java Written in Java by Linkedin and then opensourced and licenced under Apache licence. Java Written in Java by Linkedin and then opensourced and licenced under Apache licence. Highly available and durable HasWAL,works in cluster, saves data to disk by default. Highly available and durable HasWAL,works in cluster, saves data to disk by default. Highly available and durable HasWAL,works in cluster, saves data to disk by default. ️Blazing fast Sequentialwrites, zero copy. ️Blazing fast Sequentialwrites, zero copy. ️Blazing fast Sequentialwrites, zero copy.
- 27. Kafka dive in Kafka uses optimizations around Sequentialwrites to optimize disk usagewith zero copy. Kafka uses optimizations around Sequentialwrites to optimize disk usagewith zero copy. Kafka uses optimizations around Sequentialwrites to optimize disk usagewith zero copy. HasWAL log forreplication and durability. HasWAL log forreplication and durability. HasWAL log forreplication and durability. Zookeeperas separate system tracks health of the cluster. Zookeeperas separate system tracks health of the cluster. Zookeeperas separate system tracks health of the cluster. Canwork evenwithout Zookeeper. Canwork evenwithout Zookeeper. Canwork evenwithout Zookeeper. Chaos engineering shows that Kafka is highlyavailable and durable solution. Chaos engineering shows that Kafka is highlyavailable and durable solution. Chaos engineering shows that Kafka is highlyavailable and durable solution.
- 28. Debezium Debezium howto: Debezium howto: Debezium howto: Create a replication slot Create a replication slot Create a replication slot Run Debezium Java service in cluster Run Debezium Java service in cluster Run Debezium Java service in cluster Configurate itwith Groovy Configurate itwith Groovy Configurate itwith Groovy
- 29. Debezium + Pros UsesWAL directly - doesn’t create additional load toWAL(no additional data iswritten). Production ready, tested solution Lowlatency. ️ + Pros UsesWAL directly - doesn’t create additional load toWAL(no additional data iswritten). Production ready, tested solution Lowlatency. ️ + Pros UsesWAL directly - doesn’t create additional load toWAL(no additional data iswritten). Production ready, tested solution Lowlatency. ️ − Cons Howto replay data? Can you specify Log Sequence Number?What if you need to stream only a fraction ofwhat iswritten inWAL Missing Buf(protobuf on steroids) Lowflexibility and hard configurability DB Isolation. Groovywhich is not easy to use Random disconnects and need to restart.
- 30. Transactional outbox Why to use Transactional Outbox? Why to use Transactional Outbox? Why to use Transactional Outbox? Nor Kafka nor CDC can flexibly re-stream data. Nor Kafka nor CDC can flexibly re-stream data. Nor Kafka nor CDC can flexibly re-stream data. Without specific instruments you can’t remove specific events from Kafka. Without specific instruments you can’t remove specific events from Kafka. Without specific instruments you can’t remove specific events from Kafka. Replay with Kafka will require setup of additional services. Replay with Kafka will require setup of additional services. Replay with Kafka will require setup of additional services. Consistent state with the usage of Transactions. Consistent state with the usage of Transactions. Consistent state with the usage of Transactions.
- 32. ChoosingGolib confluent-kafka-go - CGO + librdkafka confluent-kafka-go - CGO + librdkafka confluent-kafka-go - CGO + librdkafka ibm/sarama ibm/sarama ibm/sarama segmentio/kafka-go segmentio/kafka-go segmentio/kafka-go
- 33. Outboxtable-WAL BatchSize-usually10. BatchSize-usually10. BatchSize-usually10. Batch Timeout-100ms. Batch Timeout-100ms. Batch Timeout-100ms. RequiedAcks-allnodesshouldconfirmmessage. RequiedAcks-allnodesshouldconfirmmessage. RequiedAcks-allnodesshouldconfirmmessage. Async-alseforsynchronouserrorhandling. Async-alseforsynchronouserrorhandling. Async-alseforsynchronouserrorhandling.
- 35. Schema registry - “Speca first” approach - speedup development. “Speca first” approach - speedup development. “Speca first” approach - speedup development. Backward compatibility support - linters. Backward compatibility support - linters. Backward compatibility support - linters. Reusable “menthal model” = simplified migration from api to stream. Reusable “menthal model” = simplified migration from api to stream. Reusable “menthal model” = simplified migration from api to stream. Client, server and models are generated for various language. Client, server and models are generated for various language. Client, server and models are generated for various language. Simplified versioning. Simplified versioning. Simplified versioning.
- 36. Taxerv1 optionwe built Update payment in Gate(kotlin). Update payment in Gate(kotlin). Update payment in Gate(kotlin). Transaction: Save payment update and create a record in Outbox. Transaction: Save payment update and create a record in Outbox. Transaction: Save payment update and create a record in Outbox. Orderstreamer(Go) - reads batch from outbox. Orderstreamer(Go) - reads batch from outbox. Orderstreamer(Go) - reads batch from outbox. Publish data to Stream. Publish data to Stream. Publish data to Stream. Update Offset in meta table. Update Offset in meta table. Update Offset in meta table.
- 37. Streamer
- 38. Meta table
- 40. v1 comparison with typical architecture + Pros We have a full transaction log that can be replayed, reworked, saved, fixed Only 1 new tech - kafka Streamer + Leaser = 200 lines of code + 800 lines of tests. It can be used as library not a service Buf/Go/PostgreSQL - everything reused - maintenance simplified. + Pros We have a full transaction log that can be replayed, reworked, saved, fixed Only 1 new tech - kafka Streamer + Leaser = 200 lines of code + 800 lines of tests. It can be used as library not a service Buf/Go/PostgreSQL - everything reused - maintenance simplified. + Pros We have a full transaction log that can be replayed, reworked, saved, fixed Only 1 new tech - kafka Streamer + Leaser = 200 lines of code + 800 lines of tests. It can be used as library not a service Buf/Go/PostgreSQL - everything reused - maintenance simplified. − Cons WAL amplification - 2x. Transactional outbox requires 1 more write to each operation High delay - 2 min for events More CPU load than just reading from WAL.
- 41. The end.
- 44. You learned about 2 min delay
- 45. Orderstreamerdelay: 2min ULIDs - doesn’t allowus to understand the commit orderof events and missing parts. ULIDs - doesn’t allowus to understand the commit orderof events and missing parts. ULIDs - doesn’t allowus to understand the commit orderof events and missing parts.
- 47. v2 Implementation Auto increment instead of ULIDwill helpyou to report and look formissing IDs. It’s more like “logical time”. Auto increment instead of ULIDwill helpyou to report and look formissing IDs. It’s more like “logical time”. Auto increment instead of ULIDwill helpyou to report and look formissing IDs. It’s more like “logical time”. Look formissing ids for, save them in meta table for2 mins and restream themwhen theyappear. Look formissing ids for, save them in meta table for2 mins and restream themwhen theyappear. Look formissing ids for, save them in meta table for2 mins and restream themwhen theyappear.
- 49. The end.
- 50. Nah,I’mkiddingx2
- 53. making online payments simple