Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

NATS Streaming - an alternative to Apache Kafka?

•
•

NATS Streaming is an alternative message broker to Apache Kafka that provides pub/sub and stream processing capabilities. It uses a commit log design where messages are appended to a log and stored even if consumed, allowing multiple consumers and time travel. While simpler than Kafka, it has limitations including lack of scalability for large channels and a smaller ecosystem of integrations compared to Kafka.

1 of 13
Download to read offline
NATS Streaming - an alternative to Apache Kafka? Anton Zadorozhniy
Agenda • Designs of message brokers • NATS project • NATS Streaming concepts • Limitations compared to Kafka • Advantages • Demo
What’s a message broker and how it’s used • Data management tool to communicate between two or more applications: • Pub/Sub, Point-to-point, Request/Response, Streams • Common uses: • Application integration • Data ingestion • Stream processing / low latency analytics
What’s inside message broker: two designs Classic design • ActiveMQ, Qpid, RabbitMQ, NATS • Features: • Routing/filtering • Re-queing • TTL/deadletter Commit log design • Kafka, Pulsar, NATS Streaming • Features: • Stream processing • Multiple consumers for the same stream • Persistence*
Classic design • Think of ArrayList • Broker tracks progress of each message, destroys message if it was consumed • Broker can re-que message if consumer failed to acknowledge consumption • Broker could implement TTL and other routing/filtering logic
Commit log design • Think of LinkedList • Only add messages to the head of list / log • Broker always stores messages, even if it was consumed multiple times (provides “time machine” capability) • Multiple consumers are cheap (no data duplication)
NATS project • Message broker written in Go with goals to be simple and fast • Maintained by Cloud Native Compute Foundation • Use cases: mobile apps, microservices and cloud native, IoT • https://nats.io • Initially in-memory message broker • Seriously fast, on good machine provides 20 mln msg/s • NATS Streaming is an extension for stream processing • Also checkout Liftbridge
NATS performance (a bit old)
NATS Streaming: concepts • Channels: persistent queues, similar to Kafka topics • Each channel is a ring buffer • Messages are written to channels even if there’s no subscription • Subscriptions: progress entity, similar to Kafka consumer groups • Messages are written to channels even if there’re no subscriptions • Sequence: position in ring buffer, similar to Kafka offset • Also allows positioning on timestamp
NATS Streaming: system architecture • Clustering: raft clusters with replicated state, only one machine is active • Fault tolerance: ability to use shared state between multiple NATS Streaming services • Partitioning: ability to assign different channels (“distribute channels”) across multiple NATS Streaming nodes to distribute load • Clustering and Partitioning/FT are incompatible!
Limitations comparing to Kafka • Replication is only available in clustering mode, with no scalability • No scalability for single channel • Instead of log controlled by retention time, NATS is using ring buffer – provides very little control on how data is deleted • Kafka persistence is actually really great (low latency), esp for large messages • Rather small ecosystem with little integrations available: • Nothing similar to Kafka Connect • No support from things like Flink or Spark Streaming
Advantages of NATS Streaming • Simplicity • Small footprint (12 MB statically linked binary, 20 MB for NATS Streaming) • Ability to use existing NATS clusters
Demo

More Related Content

NATS Streaming - an alternative to Apache Kafka?

  • 1. NATS Streaming - an alternative to Apache Kafka? Anton Zadorozhniy
  • 2. Agenda • Designs of message brokers • NATS project • NATS Streaming concepts • Limitations compared to Kafka • Advantages • Demo
  • 3. What’s a message broker and how it’s used • Data management tool to communicate between two or more applications: • Pub/Sub, Point-to-point, Request/Response, Streams • Common uses: • Application integration • Data ingestion • Stream processing / low latency analytics
  • 4. What’s inside message broker: two designs Classic design • ActiveMQ, Qpid, RabbitMQ, NATS • Features: • Routing/filtering • Re-queing • TTL/deadletter Commit log design • Kafka, Pulsar, NATS Streaming • Features: • Stream processing • Multiple consumers for the same stream • Persistence*
  • 5. Classic design • Think of ArrayList • Broker tracks progress of each message, destroys message if it was consumed • Broker can re-que message if consumer failed to acknowledge consumption • Broker could implement TTL and other routing/filtering logic
  • 6. Commit log design • Think of LinkedList • Only add messages to the head of list / log • Broker always stores messages, even if it was consumed multiple times (provides “time machine” capability) • Multiple consumers are cheap (no data duplication)
  • 7. NATS project • Message broker written in Go with goals to be simple and fast • Maintained by Cloud Native Compute Foundation • Use cases: mobile apps, microservices and cloud native, IoT • https://nats.io • Initially in-memory message broker • Seriously fast, on good machine provides 20 mln msg/s • NATS Streaming is an extension for stream processing • Also checkout Liftbridge
  • 9. NATS Streaming: concepts • Channels: persistent queues, similar to Kafka topics • Each channel is a ring buffer • Messages are written to channels even if there’s no subscription • Subscriptions: progress entity, similar to Kafka consumer groups • Messages are written to channels even if there’re no subscriptions • Sequence: position in ring buffer, similar to Kafka offset • Also allows positioning on timestamp
  • 10. NATS Streaming: system architecture • Clustering: raft clusters with replicated state, only one machine is active • Fault tolerance: ability to use shared state between multiple NATS Streaming services • Partitioning: ability to assign different channels (“distribute channels”) across multiple NATS Streaming nodes to distribute load • Clustering and Partitioning/FT are incompatible!
  • 11. Limitations comparing to Kafka • Replication is only available in clustering mode, with no scalability • No scalability for single channel • Instead of log controlled by retention time, NATS is using ring buffer – provides very little control on how data is deleted • Kafka persistence is actually really great (low latency), esp for large messages • Rather small ecosystem with little integrations available: • Nothing similar to Kafka Connect • No support from things like Flink or Spark Streaming
  • 12. Advantages of NATS Streaming • Simplicity • Small footprint (12 MB statically linked binary, 20 MB for NATS Streaming) • Ability to use existing NATS clusters
  • 13. Demo