Apache Kafka

A high throughput distributed messaging system

 What is Kakfa?

 Kafka is a distributed publish-subscribe messaging system rethought as a distributed

commit log.
 It’s designed to be
 Fast
 Scalable
 Durable
 When used in the right way and for the right use case, Kafka has unique attributes
that make it a highly attractive option for data integration.
 Publish subscribe messaging system

 Kafka maintains feeds of messages in categories called topics

 Producers are processes that publish messages to one or more topics
 Consumers are processes that subscribe to topics and process the feed of
published messages

Message Subscriber

Publisher Message Topic Message Subscriber

Message Subscriber
Kafka cluster

 Since Kafka is distributed in nature, Kafka is run as a cluster.

 A cluster is typically comprised multiple servers; each of which is called a broker.
 Communication between the clients and the servers takes place over TCP protocol

Producer Consumer
Kafka cluster
Broker 1
Topic 1 Topic 2
Producer Consumer
Broker 2
Topic 1 Topic 2

Zookeeper
Producer Consumer
Deep dive into high level abstractions
 Topic
 To balance load, a topic is divided into
multiple partitions and replicated
across brokers.
 Partitions are ordered, immutable
sequences of messages that’s
continually appended i.e. a commit log.
 The messages in the partitions are each
assigned a sequential id number called
the offset that uniquely identifies each
message within the partition.
 Distribution and partitions

 Partitions allow a topic’s log to scale beyond a size that will fit on a single server (i.e. a
broker) and act as the unit of parallelism
 The partitions of a topic are distributed over the brokers in the Kafka cluster where each
broker handles data and requests for a share of the partitions.
 For fault tolerance, each partition is replicated across a configurable number of brokers.
Distribution and fault tolerance

 Each partition has one server which acts as the "leader" and zero or more servers
which act as "followers".
 The leader handles all read and write requests for the partition while the followers
passively replicate the leader.
 If the leader fails, one of the followers will automatically become the new leader.
 Each server acts as a leader for some of its partitions and a follower for others so load
is well balanced within the cluster.
Retention

 The Kafka cluster retains all published messages—whether or not they have been
consumed—for a configurable period of time; after which it will be discarded to
free up space.
 Metadata retained on a per-consumer basis is the position of the consumer in the
log, called the offset; which is controlled by consumer.
 Normally a consumer will advance its offset linearly as it reads messages, but it can
consume messages in any order it likes.
 Kafka consumers can come and go without much impact on the cluster or on other
consumers.
Producers

 Producers publish data to the topics by assigning messages to a partition within the
topic either in a round-robin fashion or according to some semantic partition function
(say based on some key in the message).
Consumers

 Kafka offers a single consumer abstraction called consumer group that generalises
both queue and topic.
 Consumers label themselves with a consumer group name.
 Each message published to a topic is delivered to one consumer instance within each
subscribing consumer group.
 If all the consumer instances have the same consumer group, then this works just like
a traditional queue balancing load over the consumers.
 If all the consumer instances have different consumer groups, then this works like
publish-subscribe and all messages are broadcast to all consumers.
 Consumer groups

 Topics have a small number of consumer groups, one for each logical subscriber.
 Each group is composed of many consumer instances for scalability and fault tolerance.
 Ordering guarantees

 Kafka assigns partitions in a topic to consumers in a consumer group so, each partition is
consumed by exactly one consumer in the group.
 Limitation: there cannot be more consumer instances in a consumer group than partitions.
 Provides a total order over messages within a partition, not between different partitions in
a topic.
 Comaprison

Kafka JMS message broker; Rabbit MQ

A fire hose of events arriving at rate of Messages arriving at a rate of 20k+/sec
approximately 100k+/sec
‘At least once‘ processed as data is read Exactly once processed by consumers
with an offset within a partition.
Producer-centric. Doesn't have message Broker-centric. Uses the broker itself to
acknowledgements as consumers track maintain state of what's consumed (via
messages consumed. message acknowledgements)
Supports both online and batch Consumers are mostly online, and any
consumers that may be online or offline. It messages "in wait" (persistent or not) are
also supports producer message batching held opaquely.
- it's designed for holding and distributing
large volumes of messages at a very low
latency.
 Comaprison

Kafka JMS message broker; Rabbit MQ

Provides a rudimentary routing. It uses Provides rich routing capabilities with
topic for exchanges. Advanced Message Queuing Protocol’s
(AMQP) exchange, binding and queuing
model.
Makes distributed cluster explicit, by Makes the distributed cluster transparent,
forcing the producer to know it is as if it were a virtual broker
partitioning a topic's messages across
several nodes.
Preserves ordered delivery within a Almost always unordered delivery. AMQP
partition model says "one producer channel, one
exchange, one queue, one consumer
channel" is required for in-order delivery
Throttling is un-necessary

 The whole job of Kafka is to provide a "shock absorber" between the flood of
events and those who want to consume them in their own way.
Performance benchmark

 500,000 messages published per second

 22,000 messages consumed per second
 on a 2-node cluster
 with 6-disk RAID 10.
 See research.microsoft.com/en-
us/um/people/srikanth/netdb11/netdb11papers/net
db11-final12.pdf
Key benefits

 Horizontally scalable
 It’s a distributed system can be elastically and transparently expanded with no downtime
 High throughput
 High throughput is provided for both publishing and subscribing, due to disk structures
that provide constant performance even with many terabytes of stored messages
 Reliable delivery
 Persists messages on disk, and provides intra-cluster replication
 Supports large number of subscribers and automatically balances consumers in case of
failure.
Use cases

 Common use cases include

1. Stream processing, Event sourcing or a replacement for a more traditional message
broker
2. Website activity tracking - original use case for Kafka
3. Metrics collection and monitoring - centralized feeds of operational data
4. Log aggregation
Getting practical
 Download and extract Kafka

 Download the archive from

kafka.apache.org/downloads.html and
extract it
Kafka uses ZooKeeper for cluster coordination
 Kafka uses ZooKeeper; which enables
highly reliable distributed coordination so,
one needs to first start a ZooKeeper server.
 Kafka bundles a single-node ZooKeeper
instance. Single node zookeeper cluster
does NOT run a leader and a follower.
 Typically exchanged metadata include
 Kafka broker addresses
 Consumed messages offset
Common Challenges faced by distributed
system
 Outages
 Co-ordination of tasks
 Reduction of operational complexity
 Consistency and ordering guarantees

Zookeeper to rescue
 Apache Zookeeper: Definition

 Centralised service for

 Maintaining configuration information
 Naming
 Distributed synchronisation and
 providing group services.
 Apache Zookeeper: Features

 Distributed consistent data store which favours consistency over everything else.
 High availability - Tolerates a minority of an ensemble members being unavailable and
continues to function correctly.
 In an ensemble of n members where n is an odd number, the loss of (n-1)/2 members can be
tolerated.
 High performance - All the data is stored in memory and benchmarked at 50k ops/sec but
the numbers really depend on your servers and network
 Tuned for read heavy write light work load. Reads are served from the node to which a client a
connected.
 Provides strictly ordered access for data.
 Atomic write guarantees in the order they're sent to zookeeper.
 Writes are acknowledged and changes are also seen in the order they occurred.
Apache Zookeeper: Operation basics

 A cluster is an ensemble with a leaders and several followers

 Read requests are serviced from each server using its local replica but write request are forwarded to a
leader. When the leader receives a write request, it calculates what the state of the system is when the
write is to be applied and transforms this into a transaction that captures this new state.
 When zookeeper starts, it goes through a loading algorithm where by one node of the cluster is
elected to act as the leader. At any given point in time, only one node acts as a leader.
Apache Zookeeper: Operation basics

 Clients create a state full session (i.e. with heartbeats through an open socket) when they
connect to a node of an ensemble. Number of open sockets available on a zookeeper node will
limit the number of clients that connect to it.
 When a cluster member dies, clients notice a disconnect event and thus reconnect themselves
to another member of the quorum.
 Session (i.e. state of the client connected to node of an ensemble) stay alive when the client
goes down, as the session events go through the leader and gets replicated in a cluster onto
another node.
 When the leader goes down, remaining members of the cluster will re-elect a new leader using
a atomic broadcast consensus algorithm. Cluster remains unavailable only when it re-elects a
new leader.
1. Start ZooKeeper
2. Set-up a cluster with 3 brokers
2. Adjust broker
configuration
files
3. Start
kafka server
1
3. Start
kafka server
2
3. Start
kafka server
3
3. Servers 1, 2 and 3 are started and running
4. Create a kakfa topic, list topics and describe one

5. Start a producer and publish some messages

6. Start a consumer and process messages

Log directories for each of the broker instances