Documentation
Documentation
Documentation
Page 1 of 105
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
Documentation
Kafka 0.10.1 Documentation
ECOSYSTEM
CLIENTS
1. GETTING STARTED
EVENTS
CONTACT US
APACHE
1.1 Introduction
1.3 Quick Start
1.4 Ecosystem
1.5 Upgrading
2. APIS
Download
@apachekafka
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 2 of 105
6.3 Important Configs
Important Client Configs
A Production Server Configs
6.4 Java Version
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
OS
Disks and Filesystems
Application vs OS Flush Management
Linux Flush Behavior
Ext4 Notes
6.6 Monitoring
6.7 ZooKeeper
Stable Version
Operationalization
7. SECURITY
7.1 Security Overview
7.2 Encryption and Authentication using SSL
CONTACT US
APACHE
Download
Migrating Clusters
Migrating the ZooKeeper Ensemble
@apachekafka
8. KAFKA CONNECT
8.1 Overview
8.2 User Guide
8.3 Connector Development Guide
9. KAFKA STREAMS
9.1 Overview
9.2 Developer Guide
Core Concepts
Low-Level Processor API
High-Level Streams DSL
1. GETTING STARTED
1.1 Introduction
Kafka is a distributed streaming platform. What exactly does that mean?
We think of a streaming platform as having three key capabilities:
1. It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system.
2. It lets you store streams of records in a fault-tolerant way.
3. It lets you process streams of records as they occur.
What is Kafka good for?
It gets used for two broad classes of application:
1. Building real-time streaming data pipelines that reliably get data between systems or applications
2. Building real-time streaming applications that transform or react to the streams of data
To understand how Kafka does these things, let's dive in and explore Kafka's capabilities from the bottom up.
First a few concepts:
Kafka is run as a cluster on one or more servers.
The Kafka cluster stores streams of records in categories called topics.
Each record consists of a key, a value, and a timestamp.
Kafka has four core APIs:
The Producer API allows an application to publish a stream
records to one or more Kafka topics.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 3 of 105
The Consumer API allows an application to subscribe to one or
more topics and process the stream of records produced to
them.
The Streams API allows an application to act as a stream
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
streams.
The Connector API allows building and running reusable
producers or consumers that connect Kafka topics to existing
applications or data systems. For example, a connector to a
relational database might capture every change to a table.
In Kafka the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is
CLIENTS
versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in many languages
EVENTS
CONTACT US
Let's first dive into the core abstraction Kafka provides for a stream of recordsthe topic.
APACHE
A topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many
consumers that subscribe to the data written to it.
Download
For each topic, the Kafka cluster maintains a partitioned log that looks like this:
@apachekafka
Each partition is an ordered, immutable sequence of records that is continually appended toa structured commit log. The records in the partitions are each
assigned a sequential id number called the offset that uniquely identifies each record within the partition.
The Kafka cluster retains all published recordswhether or not they have been consumedusing a configurable retention period. For example, if the retention
policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space.
Kafka's performance is effectively constant with respect to data size so storing data for a long time is not a problem.
In fact, the only metadata retained on a per-consumer basis is the offset or position of that consumer in the log. This offset is controlled by the consumer:
normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in
any order it likes. For example a consumer can reset to an older offset to reprocess data from the past or skip ahead to the most recent record and start
consuming from "now".
This combination of features means that Kafka consumers are very cheapthey can come and go without much impact on the cluster or on other consumers.
For example, you can use our command line tools to "tail" the contents of any topic without changing what is consumed by any existing consumers.
The partitions in the log serve several purposes. First, they allow the log to scale beyond a size that will fit on a single server. Each individual partition must fit
on the servers that host it, but a topic may have many partitions so it can handle an arbitrary amount of data. Second they act as the unit of parallelismmore
on that in a bit.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 4 of 105
Distribution
The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of the partitions. Each
partition is replicated across a configurable number of servers for fault tolerance.
Security
Kafka Connect
Kafka Streams
PERFORMANCE
Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for
the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server
acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.
Producers
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
Producers publish data to the topics of their choice. The producer is responsible for choosing which record to assign to which partition within the topic. This
can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the
record). More on the use of partitioning in a second!
Consumers
EVENTS
CONTACT US
APACHE
Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each
subscribing consumer group. Consumer instances can be in separate processes or on separate machines.
If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.
If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.
Download
@apachekafka
A two server Kafka cluster hosting four partitions (P0-P3) with two consumer groups. Consumer group A has two consumer instances and group B has four.
More commonly, however, we have found that topics have a small number of consumer groups, one for each "logical subscriber". Each group is composed of
many consumer instances for scalability and fault tolerance. This is nothing more than publish-subscribe semantics where the subscriber is a cluster of
consumers instead of a single process.
The way consumption is implemented in Kafka is by dividing up the partitions in the log over the consumer instances so that each instance is the exclusive
consumer of a "fair share" of partitions at any point in time. This process of maintaining membership in the group is handled by the Kafka protocol
dynamically. If new instances join the group they will take over some partitions from other members of the group; if an instance dies, its partitions will be
distributed to the remaining instances.
Kafka only provides a total order over records within a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to
partition data by key is sufficient for most applications. However, if you require a total order over records this can be achieved with a topic that has only one
partition, though this will mean only one consumer process per consumer group.
Guarantees
At a high-level Kafka gives the following guarantees:
Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a record M1 is sent by the same producer
as a record M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log.
A consumer instance sees records in the order they are stored in the log.
For a topic with replication factor N, we will tolerate up to N-1 server failures without losing any records committed to the log.
More details on these guarantees are given in the design section of the documentation.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 5 of 105
multi-subscriberonce one process reads the data it's gone. Publish-subscribe allows you broadcast data to multiple processes, but has no way of scaling
processing since every message goes to every subscriber.
The consumer group concept in Kafka generalizes these two concepts. As with a queue the consumer group allows you to divide up processing over a
collection of processes (the members of the consumer group). As with publish-subscribe, Kafka allows you to broadcast messages to multiple consumer
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
groups.
The advantage of Kafka's model is that every topic has both these propertiesit can scale processing and is also multi-subscriberthere is no need to choose
one or the other.
Kafka has stronger ordering guarantees than a traditional messaging system, too.
A traditional queue retains records in-order on the server, and if multiple consumers consume from the queue then the server hands out records in the order
they are stored. However, although the server hands out records in order, the records are delivered asynchronously to consumers, so they may arrive out of
order on different consumers. This effectively means the ordering of the records is lost in the presence of parallel consumption. Messaging systems often
ECOSYSTEM
CLIENTS
EVENTS
work around this by having a notion of "exclusive consumer" that allows only one process to consume from a queue, but of course this means that there is no
parallelism in processing.
Kafka does it better. By having a notion of parallelismthe partitionwithin the topics, Kafka is able to provide both ordering guarantees and load balancing
over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is
CONTACT US
consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in
APACHE
order. Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 6 of 105
Likewise for streaming data pipelines the combination of subscription to real-time events make it possible to use Kafka for very low-latency pipelines; but the
ability to store data reliably make it possible to use it for critical data where the delivery of data must be guaranteed or for integration with offline systems
that load data only periodically or may go down for extended periods of time for maintenance. The stream processing facilities make it possible to transform
data as it arrives.
Security
Kafka Connect
For more information on the guarantees, apis, and capabilities Kafka provides see the rest of the documentation.
Kafka Streams
PERFORMANCE
Here is a description of a few of the popular use cases for Apache Kafka. For an overview of a number of these areas in action, see this blog post
POWERED BY
Messaging
PROJECT INFO
ECOSYSTEM
Kafka works well as a replacement for a more traditional message broker. Message brokers are used for a variety of reasons (to decouple processing from
data producers, to buffer unprocessed messages, etc). In comparison to most messaging systems Kafka has better throughput, built-in partitioning,
CLIENTS
replication, and fault-tolerance which makes it a good solution for large scale message processing applications.
EVENTS
In our experience messaging uses are often comparatively low-throughput, but may require low end-to-end latency and often depend on the strong durability
guarantees Kafka provides.
CONTACT US
In this domain Kafka is comparable to traditional messaging systems such as ActiveMQ or RabbitMQ.
APACHE
The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity
(page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for
@apachekafka
subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for
offline processing and reporting.
Activity tracking is often very high volume as many activity messages are generated for each user page view.
Metrics
Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of
operational data.
Log Aggregation
Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a
central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a
stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption. In comparison
to log-centric systems like Scribe or Flume, Kafka offers equally good performance, stronger durability guarantees due to replication, and much lower end-toend latency.
Stream Processing
Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then
aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing. For example, a processing pipeline for
recommending news articles might crawl article content from RSS feeds and publish it to an "articles" topic; further processing might normalize or
deduplicate this content and published the cleansed article content to a new topic; a final processing stage might attempt to recommend this content to
users. Such processing pipelines create graphs of real-time data flows based on the individual topics. Starting in 0.10.0.0, a light-weight but powerful stream
processing library called Kafka Streams is available in Apache Kafka to perform such data processing as described above. Apart from Kafka Streams,
alternative open source stream processing tools include Apache Storm and Apache Samza.
Event Sourcing
Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored
log data makes it an excellent backend for an application built in this style.
Commit Log
Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism
for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage. In this usage Kafka is similar to Apache BookKeeper
project.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 7 of 105
1.3 Quick Start
This tutorial assumes you are starting fresh and have no existing Kafka or ZooKeeper data. Since Kafka console scripts are different for Unix-based and
Windows platforms, on Windows platforms use bin\windows\ instead of bin/ , and change the script extension to .bat .
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
We can now see that topic if we run the list topic command:
Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 8 of 105
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
Operations
Security
Kafka Connect
If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see
Kafka Streams
PERFORMANCE
All of the command line tools have additional options; running the command with no arguments will display usage information documenting them in more
POWERED BY
PROJECT INFO
ECOSYSTEM
detail.
CLIENTS
than starting a few more broker instances. But just to get feel for it, let's expand our cluster to three nodes (still all on our local machine).
EVENTS
First we make a config file for each of the brokers (on Windows use the copy command instead):
CONTACT US
APACHE
Download
@apachekafka
Now edit these new files and set the following properties:
config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
The broker.id property is the unique and permanent name of each node in the cluster. We have to override the port and log directory only because we
are running these all on the same machine and we want to keep the brokers from all trying to register on the same port or overwrite each other's data.
We already have Zookeeper and our single node started, so we just need to start the two new nodes:
Okay but now that we have a cluster how can we know which broker is doing what? To see that run the "describe topics" command:
PartitionCount:1
Topic: my-replicated-topic
Partition: 0
ReplicationFactor:3
Leader: 1
Configs:
Here is an explanation of output. The first line gives a summary of all the partitions, each additional line gives information about one partition. Since we have
only one partition for this topic there is only one line.
"leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the
partitions.
"replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 9 of 105
"isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.
Note that in my example node 1 is the leader for the only partition of the topic.
We can run the same command on the original topic we created to see where it is:
Security
Kafka Connect
Kafka Streams
PartitionCount:1
Topic: test
PERFORMANCE
Partition: 0
ReplicationFactor:1
Leader: 0
Configs:
Replicas: 0
Isr: 0
POWERED BY
PROJECT INFO
So there is no surprise therethe original topic has no replicas and is on server 0, the only server in our cluster when we created it.
Let's publish a few messages to our new topic:
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
my test message 1
my test message 2
^C
Download
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
@apachekafka
...
my test message 1
my test message 2
^C
Now let's test out fault-tolerance. Broker 1 was acting as the leader so let's kill it:
7564 ttys002
0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java...
On Windows use:
java
kafka.Kafka config\ser
Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:
PartitionCount:1
Topic: my-replicated-topic
Partition: 0
ReplicationFactor:3
Leader: 2
Configs:
But the messages are still available for consumption even though the leader that took the writes originally is down:
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 10 of 105
Writing data from the console and writing it back to the console is a convenient place to start, but you'll probably want to use data from other sources or
export data from Kafka to other systems. For many systems, instead of writing custom integration code you can use Kafka Connect to import or export data.
Kafka Connect is a tool included with Kafka that imports and exports data to Kafka. It is an extensible tool that runs connectors, which implement the custom
logic for interacting with an external system. In this quickstart we'll see how to run Kafka Connect with simple connectors that import data from a file to a
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Next, we'll start two connectors running in standalone mode, which means they run in a single, local, dedicated process. We provide three configuration files
as parameters. The first is always the configuration for the Kafka Connect process, containing common configuration such as the Kafka brokers to connect to
and the serialization format for data. The remaining configuration files each specify a connector to create. These files include a unique connector name, the
connector class to instantiate, and any other configuration required by the connector.
These sample configuration files, included with Kafka, use the default local cluster configuration you started earlier and create two connectors: the first is a
source connector that reads lines from an input file and produces each to a Kafka topic and the second is a sink connector that reads messages from a Kafka
Download
@apachekafka
has started, the source connector should start reading lines from test.txt and producing them to the topic connect-test , and the sink connector
should start reading messages from the topic connect-test and write them to the file test.sink.txt . We can verify the data has been delivered
through the entire pipeline by examining the contents of the output file:
Note that the data is being stored in the Kafka topic connect-test , so we can also run a console consumer to see the data in the topic (or use custom
consumer code to process it):
The connectors continue to process data, so we can add data to the file and see it move through the pipeline:
You should see the line appear in the console consumer output and in the sink file.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 11 of 105
.countByKey("Counts")
Implementation
Operations
It implements the WordCount algorithm, which computes a word occurrence histogram from the input text. However, unlike other WordCount examples you
Security
might have seen before that operate on bounded data, the WordCount demo application behaves slightly differently because it is designed to operate on an
Kafka Connect
Kafka Streams
PERFORMANCE
infinite, unbounded stream of data. Similar to the bounded variant, it is a stateful algorithm that tracks and updates the counts of words. However, since it
must assume potentially unbounded input data, it will periodically output its current state and results while continuing to process more data because it cannot
know when it has processed "all" the input data.
We will now prepare input data to a Kafka topic, which will subsequently be processed by a Kafka Streams application.
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
> echo -e "all streams lead to kafka\nhello kafka streams\njoin kafka summit" > file-input.txt
Or on Windows:
APACHE
Next, we send this input data to the input topic named streams-file-input using the console producer (in practice, stream data will likely be flowing
Download
@apachekafka
We can now run the WordCount demo application to process the input data:
There won't be any STDOUT output except log entries as the results are continuously written back into another topic named streams-wordcount-output
Kafka. The demo will run for a few seconds and then, unlike typical stream processing applications, terminate automatically.
We can now inspect the output of the WordCount demo application by reading from its output topic:
all
lead
to
hello
streams 2
join
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Page 12 of 105
kafka
summit
Here, the first column is the Kafka message key, and the second column is the message value, both in in java.lang.String format. Note that the output
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
is actually a continuous stream of updates, where each data record (i.e. each line in the original output above) is an updated count of a single word, aka
record key such as "kafka". For multiple records with the same key, each later record is an update of the previous one.
Now you can write more input messages to the streams-file-input topic and observe additional messages added to streams-wordcount-output
reflecting updated word counts (e.g., using the console producer and the console consumer, as described above).
You can stop the console consumer via Ctrl-C.
1.4 Ecosystem
There are a plethora of tools that integrate with Kafka outside the main distribution. The ecosystem page lists many of these, including stream processing
systems, Hadoop integration, monitoring, and deployment tools.
CONTACT US
APACHE
Download
@apachekafka
The log retention time is no longer based on last modified time of the log segments. Instead it will be based on the largest timestamp of the messages in a
log segment.
The log rolling time is no longer depending on log segment create time. Instead it is now based on the timestamp in the messages. More specifically. if the
timestamp of the first message in the segment is T, the log will be rolled out when a new message has a timestamp greater than or equal to T + log.roll.ms
The open file handlers of 0.10.0 will increase by ~33% because of the addition of time index files for each segment.
The time index and offset index share the same index size configuration. Since each time index entry is 1.5x the size of offset index entry. User may need
to increase log.index.size.max.bytes to avoid potential frequent log rolling.
Due to the increased number of index files, on some brokers with large amount the log segments (e.g. >15K), the log loading process during the broker
startup could be longer. Based on our experiment, setting the num.recovery.threads.per.data.dir to one may reduce the log loading time.
Notable changes in 0.10.1.0
The new Java consumer is no longer in beta and we recommend it for all new development. The old Scala consumers are still supported, but they will be
deprecated in the next release and will be removed in a future major release.
The --new-consumer / --new.consumer switch is no longer required to use tools like MirrorMaker and the Console Consumer with the new
consumer; one simply needs to pass a Kafka broker to connect to instead of the ZooKeeper ensemble. In addition, usage of the Console Consumer with
the old consumer has been deprecated and it will be removed in a future major release.
Kafka clusters can now be uniquely identified by a cluster id. It will be automatically generated when a broker is upgraded to 0.10.1.0. The cluster id is
available via the kafka.server:type=KafkaServer,name=ClusterId metric and it is part of the Metadata response. Serializers, client interceptors and metric
reporters can receive the cluster id by implementing the ClusterResourceListener interface.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 13 of 105
The BrokerState "RunningAsController" (value 4) has been removed. Due to a bug, a broker would only be in this state briefly before transitioning out of it
and hence the impact of the removal should be minimal. The recommended way to detect if a given broker is the controller is via the
kafka.controller:type=KafkaController,name=ActiveControllerCount metric.
The new Java Consumer now allows users to search offsets by timestamp on partitions.
Security
The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls
Kafka Connect
the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration
request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can
Kafka Streams
PERFORMANCE
block on the server while the consumer is rebalancing, so we have changed its default value to just above 5 minutes. Finally, the default value of
session.timeout.ms has been adjusted down to 10 seconds, and the default value of max.poll.records has been changed to 500.
When using an Authorizer and a user doesn't have Describe authorization on a topic, the broker will no longer return TOPIC_AUTHORIZATION_FAILED
POWERED BY
PROJECT INFO
errors to requests since this leaks topic names. Instead, the UNKNOWN_TOPIC_OR_PARTITION error code will be returned. This may cause unexpected
timeouts or delays when using the producer and consumer since Kafka clients will typically retry automatically on unknown topic errors. You should
consult the client logs if you suspect this could be happening.
ECOSYSTEM
Fetch responses have a size limit by default (50 MB for consumers and 10 MB for replication). The existing per partition limits also apply (1 MB for
CLIENTS
Consumers and replicas can make progress if a message larger than the response/partition size limit is found. More concretely, if the first message in the
EVENTS
CONTACT US
consumers and replication). Note that neither of these limits is an absolute maximum as explained in the next point.
first non-empty partition of the fetch is larger than either or both limits, the message will still be returned.
Overloaded constructors were added to kafka.api.FetchRequest and kafka.javaapi.FetchRequest to allow the caller to specify the order
of the partitions (since order is significant in v3). The previously existing constructors were deprecated and the partitions are shuffled before the request is
sent to avoid starvation issues.
APACHE
New Protocol Versions
Download
@apachekafka
make progress and the order of partitions in the request is now significant.
JoinGroup v1 introduces a new field: "rebalance_timeout".
The message format in 0.10.0 includes a new timestamp field and uses relative offsets for compressed messages. The on disk message format can be
configured through log.message.format.version in the server.properties file. The default on-disk message format is 0.10.0. If a consumer client is on a version
before 0.10.0.0, it only understands message formats before 0.10.0. In this case, the broker is able to convert messages from the 0.10.0 format to an earlier
format before sending the response to the consumer on an older version. However, the broker can't use zero-copy transfer in this case. Reports from the
Kafka community on the performance impact have shown CPU utilization going from 20% before to 100% after an upgrade, which forced an immediate
upgrade of all clients to bring performance back to normal. To avoid such message conversion before consumers are upgraded to 0.10.0.0, one can set
log.message.format.version to 0.8.2 or 0.9.0 when upgrading the broker to 0.10.0.0. This way, the broker can still use zero-copy transfer to send the data to
the old consumers. Once consumers are upgraded, one can change the message format to 0.10.0 on the broker and enjoy the new message format that
includes new timestamp and improved compression. The conversion is supported to ensure compatibility and can be useful to support a few apps that have
not updated to newer clients yet, but is impractical to support all consumer traffic on even an overprovisioned cluster. Therefore, it is critical to avoid the
message conversion as much as possible when brokers have been upgraded but the majority of clients have not.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 14 of 105
For clients that are upgraded to 0.10.0.0, there is no performance impact.
Note: By setting the message format version, one certifies that all existing messages are on or below that message format version. Otherwise consumers
before 0.10.0.0 might break. In particular, after the message format is set to 0.10.0, one should not change it back to an earlier format as it may break
Security
Kafka Connect
Note: Due to the additional timestamp introduced in each message, producers sending small messages may see a message throughput degradation because
Kafka Streams
PERFORMANCE
POWERED BY
of the increased overhead. Likewise, replication now transmits an additional 8 bytes per message. If you're running close to the network capacity of your
cluster, it's possible that you'll overwhelm the network cards and see failures and performance issues due to the overload.
Note: If you have enabled compression on producers, you may notice reduced producer throughput and/or lower compression rate on the broker in some
cases. When receiving compressed messages, 0.10.0 brokers avoid recompressing the messages, which in general reduces the latency and improves the
throughput. In certain cases, however, this may reduce the batching size on the producer, which could lead to worse throughput. If this happens, users can
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
tune linger.ms and batch.size of the producer for better throughput. In addition, the producer buffer used for compressing messages with snappy is smaller
than the one used by the broker, which may have a negative impact on the compression ratio for the messages on disk. We intend to make this configurable
in a future Kafka release.
Potential breaking changes in 0.10.0.0
Starting from Kafka 0.10.0.0, the message format version in Kafka is represented as the Kafka version. For example, message format 0.9.0 refers to the
highest message version supported by Kafka 0.9.0.
Message format 0.10.0 has been introduced and it is used by default. It includes a timestamp field in the messages and relative offsets are used for
compressed messages.
ProduceRequest/Response v2 has been introduced and it is used by default to support message format 0.10.0
FetchRequest/Response v2 has been introduced and it is used by default to support message format 0.10.0
Download
MessageFormatter interface was changed from def writeTo(key: Array[Byte], value: Array[Byte], output: PrintStream)
def writeTo(consumerRecord: ConsumerRecord[Array[Byte], Array[Byte]], output: PrintStream)
@apachekafka
MessageReader interface was changed from def readMessage(): KeyedMessage[Array[Byte], Array[Byte]] to def readMessage
(): ProducerRecord[Array[Byte], Array[Byte]]
MessageFormatter's package was changed from kafka.tools to kafka.common
MessageReader's package was changed from kafka.tools to kafka.common
MirrorMakerMessageHandler no longer exposes the handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]) method as it
was never called.
The 0.7 KafkaMigrationTool is no longer packaged with Kafka. If you need to migrate from 0.7 to 0.10.0, please migrate to 0.8 first and then follow the
documented upgrade process to upgrade from 0.8 to 0.10.0.
The new consumer has standardized its APIs to accept java.util.Collection as the sequence type for method parameters. Existing code may
have to be updated to work with the 0.10.0 client library.
LZ4-compressed message handling was changed to use an interoperable framing specification (LZ4f v1.5.1). To maintain compatibility with old clients,
this change only applies to Message format 0.10.0 and later. Clients that Produce/Fetch LZ4-compressed messages using v0/v1 (Message format 0.9.0)
should continue to use the 0.9.0 framing implementation. Clients that use Produce/Fetch protocols v2 or later should use interoperable LZ4f framing. A list
of interoperable LZ4 libraries is available at http://www.lz4.org/
Notable changes in 0.10.0.0
Starting from Kafka 0.10.0.0, a new client library named Kafka Streams is available for stream processing on data stored in Kafka topics. This new client
library only works with 0.10.x and upward versioned brokers due to message format changes mentioned above. For more information please read
section.
The default value of the configuration parameter receive.buffer.bytes is now 64K for the new consumer.
The new consumer now exposes the configuration parameter exclude.internal.topics to restrict internal topics (such as the consumer offsets
topic) from accidentally being included in regular expression subscriptions. By default, it is enabled.
The old Scala producer has been deprecated. Users should migrate their code to the Java producer included in the kafka-clients JAR as soon as possible.
The new consumer API has been marked stable.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 15 of 105
Potential breaking changes in 0.9.0.0
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Download
By default, all command line tools will print all logging messages to stderr instead of stdout.
Notable changes in 0.9.0.1
@apachekafka
The new broker id generation feature can be disabled by setting broker.id.generation.enable to false.
Configuration parameter log.cleaner.enable is now true by default. This means topics with a cleanup.policy=compact will now be compacted by default,
and 128 MB of heap will be allocated to the cleaner process via log.cleaner.dedupe.buffer.size. You may want to review log.cleaner.dedupe.buffer.size and
the other log.cleaner configuration values based on your usage of compacted topics.
Default value of configuration parameter fetch.min.bytes for the new consumer is now 1 by default.
Deprecations in 0.9.0.0
Altering topic configuration from the kafka-topics.sh script (kafka.admin.TopicCommand) has been deprecated. Going forward, please use the kafkaconfigs.sh script (kafka.admin.ConfigCommand) for this functionality.
The kafka-consumer-offset-checker.sh (kafka.tools.ConsumerOffsetChecker) has been deprecated. Going forward, please use kafka-consumer-groups.sh
(kafka.admin.ConsumerGroupCommand) for this functionality.
The kafka.tools.ProducerPerformance class has been deprecated. Going forward, please use org.apache.kafka.tools.ProducerPerformance for this
functionality (kafka-producer-perf-test.sh will also be changed to use the new class).
The producer config block.on.buffer.full has been deprecated and will be removed in future release. Currently its default value has been changed to false.
The KafkaProducer will no longer throw BufferExhaustedException but instead will use max.block.ms value to block, after which it will throw a
TimeoutException. If block.on.buffer.full property is set to true explicitly, it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms
will not be honoured
2. APIS
Kafka includes four core apis:
1. The Producer API allows applications to send streams of data to topics in the Kafka cluster.
2. The Consumer API allows applications to read streams of data from topics in the Kafka cluster.
3. The Streams API allows transforming streams of data from input topics to output topics.
4. The Connect API allows implementing connectors that continually pull from some source system or application into Kafka or push from Kafka into
some sink system or application.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 16 of 105
Kafka exposes all its functionality over a language independent protocol which has clients available in many programming languages. However only the Java
clients are maintained as part of the main Kafka project, the others are available as independent open source projects. A list of non-Java clients is available
Operations
here.
Security
Kafka Connect
The Producer API allows applications to send streams of data to topics in the Kafka cluster.
Kafka Streams
Examples showing how to use the producer are given in the javadocs.
PERFORMANCE
To use the producer, you can use the following maven dependency:
POWERED BY
<dependency>
PROJECT INFO
<groupId>org.apache.kafka</groupId>
ECOSYSTEM
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
CLIENTS
EVENTS
CONTACT US
APACHE
</dependency>
Download
To use the consumer, you can use the following maven dependency:
@apachekafka
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.1.0</version>
</dependency>
3. CONFIGURATION
Kafka uses key-value pairs in the property file format for configuration. These values can be supplied either from a file or programmatically.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Kafka Streams
Page 17 of 105
3.1 Broker Configs
The essential configurations are the following:
broker.id
log.dirs
zookeeper.connect
PERFORMANCE
POWERED BY
PROJECT INFO
NAME
zookeeper.conne
ct
ECOSYSTEM
DESCRIPTION
TYPE
string
DEFAULT
VALID VALUES
IMPORTANCE
high
CLIENTS
advertised.host.n
ame
string
null
high
string
null
high
int
null
high
boolean
true
high
boolean
true
high
int
10
int
-1
high
string
producer
high
boolean
false
high
string
""
high
long
300
high
APACHE
advertised.port
auto.create.topic
s.enable
auto.leader.rebal
ance.enable
background.threa
ds
[1,...]
high
compression.type
delete.topic.enabl
e
host.name
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 18 of 105
leader.imbalance.
check.interval.sec
Operations
onds
Security
leader.imbalance.
Kafka Connect
per.broker.percen
tage
Kafka Streams
high
string
null
high
string
/tmp/kafka-logs
high
string
null
high
ecified in percentage.
log.dir
EVENTS
@apachekafka
10
CLIENTS
Download
int
ECOSYSTEM
APACHE
POWERED BY
CONTACT US
PERFORMANCE
PROJECT INFO
log.dirs
log.flush.interval.
messages
long
922337203685477
5807
[1,...]
high
ms
long
null
int
60000
high
ms is used
log.flush.offset.c
heckpoint.interva
l.ms
ecovery point
log.flush.schedul
er.interval.ms
log.retention.byte
s
log.retention.hour
s
long
[0,...]
922337203685477
high
high
5807
long
-1
high
int
168
high
int
null
high
long
null
high
int
168
[1,...]
high
int
[0,...]
high
long
null
high
long
null
high
int
1073741824
[14,...]
high
long
60000
[0,...]
high
int
1000012
[0,...]
high
log.retention.min
utes
log.retention.ms
log.roll.hours
log.roll.jitter.hour
s
log.roll.jitter.ms
log.roll.ms
log.segment.byte
s
log.segment.delet
e.delay.ms
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 19 of 105
message.max.byt
es
an receive
Operations
Security
Kafka Streams
min.insync.replic
hReplicasAfterAppend).
as
int
[1,...]
high
int
[1,...]
high
int
[1,...]
high
int
[1,...]
high
int
high
int
4096
high
short
-1
high
int
5000
[1,...]
high
int
5242880
[1,...]
high
long
600000
[1,...]
high
int
1440
[1,...]
high
int
int
50
[1,...]
high
short
[1,...]
high
int
104857600
[1,...]
high
int
9092
PROJECT INFO
CLIENTS
EVENTS
num.io.threads
CONTACT US
APACHE
num.network.thre
ads
num.recovery.thr
Download
eads.per.data.dir
@apachekafka
num.replica.fetch
ers
offset.metadata.
max.bytes
offsets.commit.re
quired.acks
offsets.commit.ti
meout.ms
offsets.load.buffe
r.size
offsets.retention.
check.interval.ms
offsets.retention.
minutes
offsets.topic.com
pression.codec
high
offsets.topic.nu
m.partitions
offsets.topic.repli
cation.factor
offsets.topic.seg
ment.bytes
port
https://kafka.apache.org/documentation
high
11/4/2016
Apache Kafka
Page 20 of 105
DEPRECATED: only used when `listeners` is not s
Implementation
Operations
Security
queued.max.requ
ests
int
500
[1,...]
high
[1,...]
high
[1,...]
high
Kafka Connect
quota.consumer.
PERFORMANCE
default
quota.producer.d
efault
CLIENTS
CONTACT US
922337203685477
5807
long
922337203685477
5807
er-second
replica.fetch.min.
bytes
APACHE
high
int
500
high
long
5000
high
long
10000
high
int
65536
high
int
30000
high
int
30000
high
int
102400
high
int
104857600
int
102400
high
boolean
true
high
int
null
high
int
6000
high
boolean
false
high
boolean
true
medium
replica.fetch.wait.
Download
long
PROJECT INFO
EVENTS
POWERED BY
ECOSYSTEM
max.ms
@apachekafka
replica.high.water
mark.checkpoint.i
nterval.ms
replica.lag.time.m
ax.ms
replica.socket.rec
eive.buffer.bytes
replica.socket.tim
eout.ms
request.timeout.
ms
socket.receive.bu
ffer.bytes
socket.request.m
ax.bytes
st
socket.send.buffe
r.bytes
unclean.leader.el
ection.enable
zookeeper.conne
ction.timeout.ms
zookeeper.sessio
n.timeout.ms
zookeeper.set.acl
broker.id.generati
on.enable
[1,...]
high
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 21 of 105
broker.rack
connections.max.
Kafka Connect
idle.ms
Kafka Streams
controlled.shutdo
PROJECT INFO
ECOSYSTEM
CLIENTS
wn.enable
controlled.shutdo
wn.max.retries
APACHE
Download
long
600000
medium
boolean
true
medium
int
medium
long
5000
medium
int
30000
medium
int
medium
int
1000
medium
int
300000
medium
int
6000
medium
string
0.10.1-IV2
medium
long
15000
long
134217728
medium
long
86400000
medium
boolean
true
medium
double
0.9
medium
int
524288
re than this
Enable controlled shutdown of the server
Controlled shutdown can fail for multiple reason
s. This determines the number of retries when su
ch failure happens
Before each retry, the system needs time to reco
ver from the state that caused the previous failur
wn.retry.backoff.
ms
controller.socket.
timeout.ms
nels
default.replicatio
n.factor
ed topics
fetch.purgatory.p
urge.interval.requ
@apachekafka
medium
controlled.shutdo
EVENTS
CONTACT US
null
Security
POWERED BY
string
Operations
PERFORMANCE
ests
group.max.sessio
n.timeout.ms
group.min.sessio
n.timeout.ms
inter.broker.proto
col.version
log.cleaner.backo
ff.ms
ogs to clean
log.cleaner.dedup
e.buffer.size
log.cleaner.delet
e.retention.ms
[0,...]
medium
log.cleaner.io.buf
fer.load.factor
log.cleaner.io.buf
fer.size
log.cleaner.io.ma
x.bytes.per.secon
of its read and write i/o will be less than this valu
e on average
https://kafka.apache.org/documentation
double
1.79769313486231
57E308
[0,...]
medium
medium
11/4/2016
Apache Kafka
Implementation
Operations
Security
Page 22 of 105
log.cleaner.min.cl
eanable.ratio
log.cleaner.min.c
ompaction.lag.m
log.cleaner.thread
g cleaning
double
0.5
medium
long
medium
int
list
[delete]
int
4096
[0,...]
medium
int
10485760
[4,...]
medium
string
0.10.1-IV2
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
CLIENTS
EVENTS
[compact, delet
e]
medium
mpact"
PROJECT INFO
ECOSYSTEM
medium
[0,...]
log.index.interval.
bytes
fset index
log.index.size.ma
x.bytes
CONTACT US
APACHE
mat.version
medium
@apachekafka
long
922337203685477
5807
[0,...]
medium
log.message.time
stamp.type
[CreateTime, Log
string
CreateTime
boolean
false
long
300000
[1,...]
medium
int
2147483647
[1,...]
medium
string
""
int
AppendTime]
medium
Time`
Should pre allocate file when create new segmen
log.preallocate
medium
k.interval.ms
max.connections.
per.ip
max.connections.
per.ip.overrides
num.partitions
principal.builder.c
lass
ry.purge.interval.r
equests
replica.fetch.back
off.ms
error occurs.
replica.fetch.max.
bytes
https://kafka.apache.org/documentation
[1,...]
medium
class org.apache.k
class
medium
afka.common.secu
medium
rity.auth.DefaultPri
ncipalBuilder
int
1000
medium
int
1000
[0,...]
medium
int
1048576
[0,...]
medium
11/4/2016
Apache Kafka
Page 23 of 105
maximum, if the first message in the first non-e
Implementation
Operations
Kafka Connect
Kafka Streams
POWERED BY
PROJECT INFO
replica.fetch.resp
onse.max.bytes
ECOSYSTEM
CONTACT US
APACHE
int
10485760
[0,...]
medium
int
1000
[0,...]
medium
list
[GSSAPI]
medium
string
/usr/bin/kinit
medium
long
60000
medium
list
[DEFAULT]
medium
string
null
medium
double
0.05
medium
double
0.8
medium
string
GSSAPI
medium
string
PLAINTEXT
medium
list
null
medium
string
none
reserved.broker.
max.id
sasl.enabled.mec
hanisms
Download
sasl.kerberos.kini
@apachekafka
CLIENTS
EVENTS
t.cmd
sasl.kerberos.mi
n.time.before.relo
gin
sasl.kerberos.prin
cipal.to.local.rule
sasl.kerberos.ser
vice.name
sasl.kerberos.tick
et.renew.jitter
l time.
sasl.kerberos.tick
et.renew.window.
factor
sasl.mechanism.i
nter.broker.protoc
ol
security.inter.brok
er.protocol
ssl.cipher.suites
ssl.client.auth
[required, reques
medium
ted, none]
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 24 of 105
Implementation
ssl.client.auth=required If set to r
equired client authentication is required.
Operations
ssl.client.auth=requested This me
ans client authentication is optional. unlike re
Security
Kafka Connect
bout itself
Kafka Streams
POWERED BY
PROJECT INFO
ECOSYSTEM
ssl.enabled.proto
cols
s.
ssl.key.password
CLIENTS
EVENTS
list
[TLSv1.2, TLSv1.1,
medium
TLSv1]
password
null
medium
string
SunX509
medium
string
null
medium
password
null
medium
string
JKS
medium
string
TLS
medium
string
null
medium
string
PKIX
medium
string
null
medium
password
null
medium
string
JKS
medium
string
""
low
list
[]
low
int
[1,...]
low
long
30000
[1,...]
low
int
11
[1,...]
low
int
[1,...]
low
lgorithm
ssl.keystore.locat
ion
Download
ssl.keymanager.a
CONTACT US
APACHE
PERFORMANCE
ssl.keystore.pass
word
@apachekafka
ssl.keystore.type
ssl.protocol
ssl.provider
ssl.trustmanager.
algorithm
ssl.truststore.loca
tion
ssl.truststore.pas
sword
ssl.truststore.typ
e
authorizer.class.n
ame
orization
A list of classes to use as metrics reporters. Impl
ementing the MetricReporter interface all
metric.reporters
metrics.num.sam
ples
metrics.
metrics.sample.w
indow.ms
ed over.
quota.window.nu
lient quotas
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 25 of 105
quota.window.siz
e.seconds
replication.quota.
Security
window.num
eplication quotas
Kafka Connect
replication.quota.
window.size.seco
Kafka Streams
nds
PERFORMANCE
ssl.endpoint.ident
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
ification.algorith
m
ssl.secure.rando
m.implementatio
n
zookeeper.sync.ti
me.ms
int
11
[1,...]
low
int
[1,...]
low
string
null
low
string
null
low
int
2000
low
More details about broker configuration can be found in the scala class kafka.server.KafkaConfig .
Topic-level configuration Configurations pertinent to topics have both a server default as well an optional per-topic override. If no per-topic configuration is
given the server default is used. The override can be set at topic creation time by giving one or more --config options. This example creates a topic
Download
named my-topic with a custom max message size and flush rate:
@apachekafka
Overrides can also be changed or set later using the alter configs command. This example updates the max message size for my-topic:
> bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name my-topic --alter --add-conf
The following are the topic-level configurations. The server's default configuration for this property is given under the Server Default Property heading. A given
server default config value only applies to a topic if it does not have an explicit topic config override.
SERVER
NAME
DESCRIPTION
TYPE
DEFAULT
VALID VALUES
DEFAULT
IMPORTANCE
PROPERTY
A string that is either "delete" or
"compact". This string designate
s the retention policy to use on o
ld log segments. The default poli
cleanup.policy
list
[delete]
[compact, delete]
string
producer
log.cleanup.poli
cy
medium
[uncompressed,
compression.typ
producer]
https://kafka.apache.org/documentation
medium
11/4/2016
Apache Kafka
Page 26 of 105
lz4). It additionally accepts 'unco
Implementation
Operations
Kafka Connect
oducer.
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
delete.retention.
ms
ECOSYSTEM
long
86400000
[0,...]
long
60000
[0,...]
log.cleaner.delet
e.retention.ms
medium
CLIENTS
EVENTS
CONTACT US
file.delete.delay.
ms
APACHE
log.segment.del
ete.delay.ms
medium
Download
@apachekafka
long
922337203685
4775807
[0,...]
log.flush.interva
l.messages
medium
long
922337203685
4775807
[0,...]
log.flush.interva
l.ms
medium
kafka.server.Thr
list
[]
tionId]:[BrokerId]:... or alternative
ottledReplicaList
Validator$@59d5
7c39
follower.replicati
on.throttled.repli
medium
cas
tes
int
4096
list
[]
[0,...]
log.index.interva
l.bytes
medium
https://kafka.apache.org/documentation
medium
11/4/2016
Apache Kafka
Implementation
Operations
Page 27 of 105
leader.replicatio
kafka.server.Thr
leader.replicatio
n.throttled.replic
ottledReplicaList
n.throttled.replic
as
Validator$@59d5
as
7c39
Security
rm [PartitionId]:[BrokerId],[Partiti
onId]:[BrokerId]:... or alternativel
Kafka Connect
PERFORMANCE
POWERED BY
PROJECT INFO
ytes
int
1000012
string
0.10.1-IV2
[0,...]
message.max.b
ytes
medium
CLIENTS
EVENTS
CONTACT US
APACHE
message.forma
t.version
log.message.for
mat.version
medium
@apachekafka
message. If message.timestam
amp.difference.
p.type=CreateTime, a message
max.ms
long
922337203685
4775807
log.message.tim
[0,...]
estamp.differen
medium
ce.max.ms
log.message.tim
string
CreateTime
double
0.5
[0,...,1]
long
[0,...]
estamp.type
medium
or `LogAppendTime`
This configuration controls how
frequently the log compactor wil
l attempt to clean the log (assu
ming log compaction is enable
d). By default we will avoid clean
ing a log where more than 50% o
min.cleanable.di
rty.ratio
log.cleaner.min.
cleanable.ratio
medium
lag.ms
log.cleaner.min.
compaction.lag.
medium
ms
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 28 of 105
min.insync.repli
cas
int
[1,...]
min.insync.replic
medium
as
Operations
Kafka Connect
PERFORMANCE
ReplicasAfterAppend).
When used together, min.insync.
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
preallocate
Download
boolean
false
log.preallocate
medium
long
-1
long
604800000
int
1073741824
[14,...]
int
10485760
[0,...]
long
[0,...]
log.roll.jitter.ms
medium
long
604800000
[0,...]
log.roll.ms
medium
boolean
true
unclean.leader.el
medium
@apachekafka
log.retention.byt
es
medium
log.retention.ms
medium
log.segment.byt
es
medium
log.index.size.m
ax.bytes
medium
segment.ms
unclean.leader.e
lection.enable
https://kafka.apache.org/documentation
ection.enable
11/4/2016
Apache Kafka
Page 29 of 105
ed as leader as a last resort, eve
Implementation
Operations
Security
data loss
Kafka Connect
Kafka Streams
PERFORMANCE
NAME
DESCRIPTION
TYPE
DEFAULT
VALID VALUES
IMPORTANCE
POWERED BY
A list of host/port pairs to use for establishing the
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
bootstrap.server
s
list
high
class
high
class
high
CONTACT US
APACHE
value.serializer
acks
o -1.
string
[all, -1, 0, 1]
high
long
33554432
[0,...]
high
buffer.memory
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 30 of 105
und since not all memory the producer uses is use
Implementation
Operations
Kafka Connect
Kafka Streams
PERFORMANCE
compression.typ
e
POWERED BY
string
none
high
int
password
null
high
string
null
high
password
null
high
string
null
high
password
null
high
int
16384
string
""
medium
long
540000
medium
long
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
retries
max.in.flight.requests.per.connectio
n to 1 will potentially change the ordering of reco
[0,...,214748364
7]
high
APACHE
tition, and the first fails and is retried but the seco
nd succeeds, then the records in the second batch
may appear first.
Download
@apachekafka
ssl.key.passwor
ssl.keystore.loc
ation
ssl.keystore.pas
sword
ssl.truststore.lo
cation
ssl.truststore.pa
ssword
[0,...]
medium
connections.ma
x.idle.ms
linger.ms
https://kafka.apache.org/documentation
[0,...]
medium
11/4/2016
Apache Kafka
Page 31 of 105
The producer groups together any records that arr
Implementation
Operations
Kafka Connect
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Download
long
60000
[0,...]
medium
int
1048576
[0,...]
medium
Note that the server has its own cap on record siz
e which may be different from this. This setting wil
l limit the number of record batches the producer
will send in a single request to avoid sending huge
requests.
class org.apach
partitioner.class
e.kafka.clients.pr
class
oducer.internals.
medium
DefaultPartitione
r
receive.buffer.by
tes
int
32768
[-1,...]
medium
int
30000
[0,...]
medium
string
null
medium
string
GSSAPI
medium
string
PLAINTEXT
medium
int
131072
request.timeout.
ms
sasl.kerberos.se
rvice.name
sasl.mechanism
security.protoco
l
https://kafka.apache.org/documentation
[-1,...]
medium
11/4/2016
Apache Kafka
Implementation
Page 32 of 105
send.buffer.byte
Operations
ssl.enabled.prot
Security
ocols
list
[TLSv1.2, TLSv1.
medium
1, TLSv1]
Kafka Connect
Kafka Streams
ssl.keystore.typ
l for client.
PERFORMANCE
JKS
medium
string
TLS
medium
string
null
medium
string
JKS
medium
int
30000
boolean
false
low
list
null
low
int
[1,...]
low
long
60000
[0,...]
low
long
300000
[0,...]
low
POWERED BY
PROJECT INFO
string
ssl.protocol
ECOSYSTEM
CLIENTS
ssl.provider
EVENTS
CONTACT US
APACHE
ssl.truststore.ty
pe
Download
@apachekafka
[0,...]
medium
ull
es
max.in.flight.req
uests.per.conne
ction
metadata.fetch.t
imeout.ms
metadata.max.a
ge.ms
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 33 of 105
any partition leadership changes to proactively dis
Implementation
Operations
Security
metric.reporters
Kafka Connect
POWERED BY
PROJECT INFO
int
[1,...]
low
long
30000
[0,...]
low
long
50
[0,...]
low
long
100
[0,...]
low
string
/usr/bin/kinit
low
long
60000
low
double
0.05
low
double
0.8
low
list
null
low
string
null
low
string
SunX509
low
string
null
low
string
PKIX
low
metrics.num.sa
mples
etrics.
metrics.sample.
window.ms
d over.
The amount of time to wait before attempting to r
reconnect.backo
ff.ms
EVENTS
CONTACT US
APACHE
low
ECOSYSTEM
CLIENTS
[]
Kafka Streams
PERFORMANCE
list
Download
@apachekafka
sasl.kerberos.ki
nit.cmd
sasl.kerberos.mi
n.time.before.rel
ogin
sasl.kerberos.tic
ket.renew.jitter
ime.
sasl.kerberos.tic
ket.renew.windo
w.factor
ssl.cipher.suites
ssl.endpoint.ide
ntification.algori
thm
ssl.keymanager.
algorithm
ssl.secure.rando
m.implementati
on
ssl.trustmanage
r.algorithm
Machine.
For those interested in the legacy Scala producer configs, information can be found here.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 34 of 105
3.3.1 New Consumer Configs
Below is the configuration for the new consumer:
Security
Kafka Connect
NAME
DEFAULT
VALID VALUES
IMPORTANCE
PERFORMANCE
POWERED BY
ECOSYSTEM
list
high
class
high
class
high
CLIENTS
EVENTS
CONTACT US
APACHE
TYPE
Kafka Streams
PROJECT INFO
DESCRIPTION
key.deserializer
value.deserializer
Download
@apachekafka
ble the request will wait for that much data to accu
mulate before answering the request. The default s
etting of 1 byte means that fetch requests are ans
fetch.min.bytes
int
[0,...]
string
""
high
int
3000
high
int
1048576
int
10000
high
heartbeat.interva
l.ms
max.partition.fet
ch.bytes
[0,...]
high
session.timeout.
ms
high
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 35 of 105
rom the group and initiate a rebalance. Note that th
Implementation
Operations
Security
on.timeout.ms .
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ssl.key.password
ssl.keystore.locat
ion
ssl.keystore.pass
ECOSYSTEM
word
CLIENTS
ssl.truststore.loc
EVENTS
ation
ssl.truststore.pas
CONTACT US
sword
APACHE
password
null
high
string
null
high
password
null
high
string
null
high
password
null
high
string
latest
long
540000
medium
boolean
true
medium
boolean
true
medium
int
52428800
[0,...]
medium
int
300000
[1,...]
medium
int
500
[1,...]
medium
Download
@apachekafka
auto.offset.reset
[latest, earliest,
none]
medium
x.idle.ms
enable.auto.com
mit
exclude.internal.t
opics
fetch.max.bytes
max.poll.interval.
ms
max.poll.records
partition.assignm
ent.strategy
anagement is used
https://kafka.apache.org/documentation
[class org.apache.
list
kafka.clients.cons
umer.RangeAssig
medium
nor]
11/4/2016
Apache Kafka
Implementation
Page 36 of 105
receive.buffer.byt
es
int
65536
[-1,...]
medium
int
305000
[0,...]
medium
string
null
medium
string
GSSAPI
medium
string
PLAINTEXT
medium
int
131072
Operations
Security
Kafka Connect
request.timeout.
ms
Kafka Streams
PERFORMANCE
POWERED BY
sasl.kerberos.ser
vice.name
PROJECT INFO
ECOSYSTEM
sasl.mechanism
CLIENTS
security.protocol
SL_SSL.
CONTACT US
APACHE
[-1,...]
medium
ssl.enabled.proto
cols
ssl.keystore.type
list
[TLSv1.2, TLSv1.1,
medium
TLSv1]
string
JKS
medium
string
TLS
medium
string
null
medium
string
JKS
medium
int
5000
boolean
true
low
string
""
low
int
500
list
null
long
300000
ssl.provider
ssl.truststore.typ
e
auto.commit.inte
rval.ms
[0,...]
low
check.crcs
client.id
fetch.max.wait.m
[0,...]
low
given by fetch.min.bytes.
A list of classes to use as interceptors. Implementi
interceptor.class
es
low
metadata.max.ag
e.ms
[0,...]
low
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 37 of 105
any partition leadership changes to proactively dis
Implementation
Operations
Security
metric.reporters
Kafka Connect
list
[]
int
[1,...]
low
long
30000
[0,...]
low
long
50
[0,...]
low
long
100
[0,...]
low
string
/usr/bin/kinit
low
long
60000
low
double
0.05
low
double
0.8
low
list
null
low
string
null
low
string
SunX509
low
string
null
low
string
PKIX
low
low
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
metrics.num.sam
ples
etrics.
metrics.sample.
window.ms
over.
The amount of time to wait before attempting to re
ECOSYSTEM
reconnect.backof
CLIENTS
f.ms
EVENTS
CONTACT US
APACHE
sasl.kerberos.kini
Download
t.cmd
sasl.kerberos.mi
@apachekafka
n.time.before.rel
ogin
sasl.kerberos.tic
ket.renew.jitter
me.
sasl.kerberos.tic
ket.renew.windo
w.factor
ssl.cipher.suites
ssl.endpoint.iden
tification.algorith
m
ssl.keymanager.a
lgorithm
ssl.secure.rando
m.implementatio
n
ssl.trustmanager.
algorithm
achine.
group.id
zookeeper.connect
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 38 of 105
PROPERTY
DEFAULT
DESCRIPTION
A string that uniquely identifies the group of consumer processes to which this consumer belongs. By setting t
group.id
he same group id multiple processes indicate that they are all part of the same consumer group.
Security
Specifies the ZooKeeper connection string in the form hostname:port where host and port are the host an
d port of a ZooKeeper server. To allow connecting through other ZooKeeper nodes when that ZooKeeper machi
Kafka Connect
ne is down you can also specify multiple hosts in the form hostname1:port1,hostname2:port2,hostn
Kafka Streams
PERFORMANCE
ame3:port3 .
zookeeper.connec
t
The server may also have a ZooKeeper chroot path as part of its ZooKeeper connection string which puts its d
ata under some path in the global ZooKeeper namespace. If so the consumer should use the same chroot path
POWERED BY
in its connection string. For example to give a chroot path of /chroot/path you would give the connection
PROJECT INFO
string as hostname1:port1,hostname2:port2,hostname3:port3/chroot/path .
ECOSYSTEM
CLIENTS
consumer.id
null
EVENTS
socket.timeout.ms
30 * 1000
The socket timeout for network requests. The actual timeout set will be max.fetch.wait + socket.timeout.ms.
CONTACT US
socket.receive.buf
64 * 1024
APACHE
fer.bytes
The number of bytes of messages to attempt to fetch for each topic-partition in each fetch request. These byte
fetch.message.ma
Download
@apachekafka
x.bytes
1024 * 1024
s will be read into memory for each partition, so this helps control the memory used by the consumer. The fetc
h request size must be at least as large as the maximum message size the server allows or else it is possible f
or the producer to send messages larger than the consumer can fetch.
num.consumer.fet
chers
auto.commit.enabl
e
auto.commit.interv
al.ms
queued.max.mess
age.chunks
rebalance.max.retr
ies
true
60 * 1000
fetch.wait.max.ms
100
refresh.leader.bac
koff.ms
tted offset will be used when the process fails as the position from which the new consumer will begin.
The frequency in ms that the consumer offsets are committed to zookeeper.
Max number of message chunks buffered for consumption. Each chunk can be up to fetch.message.max.byte
s.
partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalan
ce will fail and retry. This setting controls the maximum number of attempts before giving up.
ms
If true, periodically commit to ZooKeeper the offset of messages already fetched by the consumer. This commi
When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign
4
fetch.min.bytes
rebalance.backoff.
2000
200
The minimum amount of data the server should return for a fetch request. If insufficient data is available the re
quest will wait for that much data to accumulate before answering the request.
The maximum amount of time the server will block before answering the fetch request if there isn't sufficient d
ata to immediately satisfy fetch.min.bytes
Backoff time between retries during rebalance. If not set explicitly, the value in zookeeper.sync.time.ms is use
d.
Backoff time to wait before trying to determine the leader of a partition that has just lost its leader.
largest
consumer.timeout.
ms
exclude.internal.to
pics
client.id
zookeeper.sessio
n.timeout.ms
zookeeper.connec
tion.timeout.ms
-1
true
group id value
6000
Throw a timeout exception to the consumer if no message is available for consumption after the specified inte
rval
Whether messages from internal topics (such as offsets) should be exposed to the consumer.
The client id is a user-specified string sent in each request to help trace calls. It should logically identify the app
lication making the request.
ZooKeeper session timeout. If the consumer fails to heartbeat to ZooKeeper for this period of time it is conside
red dead and a rebalance will occur.
6000
The max time that the client waits while establishing a connection to zookeeper.
2000
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 39 of 105
zookeeper.sync.ti
me.ms
Operations
offsets.storage
Security
offsets.channel.ba
Kafka Connect
ckoff.ms
Kafka Streams
offsets.channel.so
cket.timeout.ms
zookeeper
1000
The backoff period when reconnecting the offsets channel or retrying failed offset fetch/commit requests.
10000
PERFORMANCE
POWERED BY
merMetadata requests that are used to query for the offset manager.
Retry the offset commit up to this many times on failure. This retry count only applies to offset commits during
offsets.commit.m
ax.retries
PROJECT INFO
shut-down. It does not apply to commits originating from the auto-commit thread. It also does not apply to atte
mpts to query for the offset coordinator before committing offsets. i.e., if a consumer metadata request fails f
or any reason, it will be retried and that retry does not count toward this limit.
ECOSYSTEM
CLIENTS
Socket timeout when reading responses for offset fetch/commit requests. This timeout is also used for Consu
If you are using "kafka" as offsets.storage, you can dual commit offsets to ZooKeeper (in addition to Kafka). Th
dual.commit.enabl
ed
true
is is required during migration from zookeeper-based offset storage to kafka-based offset storage. With respec
t to any given consumer group, it is safe to turn this off after all instances within that group have been migrated
EVENTS
to the new version that commits offsets to the broker (instead of directly to ZooKeeper).
CONTACT US
Select between the "range" or "roundrobin" strategy for assigning partitions to consumer streams.
APACHE
The round-robin partition assignor lays out all the available partitions and all the available consumer threads. It
then proceeds to do a round-robin assignment from partition to consumer thread. If the subscriptions of all con
sumer instances are identical, then the partitions will be uniformly distributed. (i.e., the partition ownership cou
Download
partition.assignme
nt.strategy
nts will be within a delta of exactly one across all consumer threads.) Round-robin assignment is permitted onl
range
y if: (a) Every topic has the same number of streams within a consumer instance (b) The set of subscribed topi
cs is identical for every consumer instance within the group.
@apachekafka
Range partitioning works on a per-topic basis. For each topic, we lay out the available partitions in numeric ord
er and the consumer threads in lexicographic order. We then divide the number of partitions by the total numbe
r of consumer streams (threads) to determine the number of partitions to assign to each consumer. If it does n
ot evenly divide, then the first few consumers will have one extra partition.
More details about consumer configuration can be found in the scala class kafka.consumer.ConsumerConfig .
NAME
config.storage.to
pic
group.id
DESCRIPTION
TYPE
string
high
string
high
class
high
string
high
string
high
class
high
class
low
DEFAULT
VALID VALUES
IMPORTANCE
offset.storage.to
pic
status.storage.to
pic
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 40 of 105
internal.key.conv
erter
Operations
Kafka Connect
PERFORMANCE
POWERED BY
mentation.
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
internal.value.con
verter
CONTACT US
class
low
APACHE
Download
@apachekafka
list
[localhost:9092]
high
int
3000
high
int
60000
high
int
10000
high
password
null
high
string
null
high
rebalance.timeou
t.ms
session.timeout.
ms
ssl.key.password
ssl.keystore.locat
ion
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 41 of 105
ssl.keystore.pass
The store password for the key store file. This is opti
word
null
high
string
null
high
password
null
high
long
540000
medium
int
32768
[0,...]
medium
int
40000
[0,...]
medium
string
null
medium
string
GSSAPI
medium
string
PLAINTEXT
medium
int
131072
n is configured.
Operations
ssl.truststore.loc
Security
password
ation
Kafka Connect
ssl.truststore.pas
Kafka Streams
sword
PERFORMANCE
connections.ma
x.idle.ms
POWERED BY
PROJECT INFO
receive.buffer.byt
es
ECOSYSTEM
request.timeout.
ms
CONTACT US
APACHE
CLIENTS
EVENTS
sasl.kerberos.ser
vice.name
Download
sasl.mechanism
@apachekafka
send.buffer.bytes
[0,...]
medium
ill be used.
ssl.enabled.proto
cols
ssl.keystore.type
list
[TLSv1.2, TLSv1.
1, TLSv1]
medium
string
JKS
medium
string
TLS
medium
string
null
medium
string
JKS
medium
int
3000
medium
int
300000
medium
string
""
low
string
""
low
ssl.provider
ssl.truststore.typ
e
worker.sync.time
out.ms
worker.unsync.ba
ckoff.ms
access.control.all
ow.methods
access.control.all
ow.origin
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 42 of 105
ld be permitted to access the API, or '*' to allow acce
Implementation
Operations
Security
Kafka Connect
client.id
string
""
low
long
300000
list
[]
int
[1,...]
low
long
30000
[0,...]
low
long
60000
low
long
5000
low
long
50
string
null
low
int
null
low
string
null
low
int
8083
low
long
100
string
/usr/bin/kinit
low
long
60000
low
double
0.05
low
double
0.8
low
list
null
low
Kafka Streams
est logging.
PERFORMANCE
POWERED BY
PROJECT INFO
e.ms
low
ECOSYSTEM
CLIENTS
[0,...]
low
EVENTS
er JMX statistics.
CONTACT US
APACHE
Download
@apachekafka
metrics.num.sam
ples
ics.
metrics.sample.
window.ms
ver.
offset.flush.interv
al.ms
offset.flush.time
out.ms
reconnect.backof
f.ms
[0,...]
low
ost.name
rest.advertised.p
ort
rest.host.name
rest.port
retry.backoff.ms
[0,...]
low
me failure scenarios.
sasl.kerberos.kini
t.cmd
sasl.kerberos.mi
n.time.before.rel
ogin
sasl.kerberos.tick
et.renew.jitter
e.
sasl.kerberos.tick
et.renew.window.
factor
ssl.cipher.suites
ported.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 43 of 105
ssl.endpoint.iden
tification.algorith
Operations
Security
ssl.keymanager.a
Kafka Connect
lgorithm
Kafka Streams
ssl.secure.rando
PERFORMANCE
m.implementatio
n
POWERED BY
PROJECT INFO
ssl.trustmanager.
algorithm
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
task.shutdown.gr
aceful.timeout.m
s
string
null
low
string
SunX509
low
string
null
low
string
PKIX
low
long
5000
low
Download
NAME
@apachekafka
DESCRIPTION
TYPE
DEFAULT
VALID VALUES
IMPORTANCE
string
high
list
high
client.id
string
""
high
string
""
high
ct
ement.
class org.apache.k
key.serde
class
afka.common.seria
lization.Serdes$Byt
medium
eArraySerde
class org.apache.k
partition.grouper
class
afka.streams.proce
ssor.DefaultPartitio
medium
nGrouper
The replication factor for change log topics and r
replication.factor
int
medium
g application.
state.dir
https://kafka.apache.org/documentation
string
class
/tmp/kafka-stream
s
medium
medium
11/4/2016
Apache Kafka
Implementation
Page 44 of 105
timestamp.extrac
tor
class org.apache.k
TimestampExtractor interface.
afka.streams.proce
ssor.ConsumerRec
Operations
ordTimestampExtra
Security
ctor
Kafka Connect
Kafka Streams
class org.apache.k
value.serde
class
PERFORMANCE
POWERED BY
PROJECT INFO
windowstore.cha
ngelog.additional.
retention.ms
APACHE
Download
long
86400000
medium
string
""
low
int
1000
low
long
10485760
long
30000
low
list
[]
low
int
[1,...]
low
long
30000
[0,...]
low
int
low
int
low
long
100
low
class
null
low
long
60000
low
EVENTS
CONTACT US
medium
lization.Serdes$Byt
eArraySerde
ECOSYSTEM
CLIENTS
afka.common.seria
buffered.records.
per.partition
rtition.
cache.max.bytes.
buffering
commit.interval.
ms
he processor.
[0,...]
low
@apachekafka
metrics.num.sam
ples
metrics.
metrics.sample.w
indow.ms
d over.
num.standby.repli
cas
num.stream.threa
ds
sing.
poll.ms
rocksdb.config.se
tter
he RocksDBConfigSetter interface
state.cleanup.del
ay.ms
4. DESIGN
4.1 Motivation
We designed Kafka to be able to act as a unified platform for handling all the real-time data feeds a large company might have. To do this we had to think
through a fairly broad set of use cases.
It would have to have high-throughput to support high volume event streams such as real-time log aggregation.
It would need to deal gracefully with large data backlogs to be able to support periodic data loads from offline systems.
It also meant the system would have to handle low-latency delivery to handle more traditional messaging use-cases.
We wanted to support partitioned, distributed, real-time processing of these feeds to create new, derived feeds. This motivated our partitioning and consumer
model.
Finally in cases where the stream is fed into other data systems for serving, we knew the system would have to be able to guarantee fault-tolerance in the
presence of machine failures.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Page 45 of 105
Supporting these uses led us to a design with a number of unique elements, more akin to a database log than a traditional messaging system. We will outline
some elements of the design in the following sections.
4.2 Persistence
Don't fear the filesystem!
Kafka relies heavily on the filesystem for storing and caching messages. There is a general perception that "disks are slow" which makes people skeptical
that a persistent structure can offer competitive performance. In fact disks are both much slower and much faster than people expect depending on how they
are used; and a properly designed disk structure can often be as fast as the network.
The key fact about disk performance is that the throughput of hard drives has been diverging from the latency of a disk seek for the last decade. As a result
the performance of linear writes on a JBOD configuration with six 7200rpm SATA RAID-5 array is about 600MB/sec but the performance of random writes is
only about 100k/seca difference of over 6000X. These linear reads and writes are the most predictable of all usage patterns, and are heavily optimized by
the operating system. A modern operating system provides read-ahead and write-behind techniques that prefetch data in large block multiples and group
smaller logical writes into large physical writes. A further discussion of this issue can be found in this ACM Queue article; they actually find that
disk access can in some cases be faster than random memory access!
To compensate for this performance divergence, modern operating systems have become increasingly aggressive in their use of main memory for disk
caching. A modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed. All disk reads and
writes will go through this unified cache. This feature cannot easily be turned off without using direct I/O, so even if a process maintains an in-process cache
of the data, this data will likely be duplicated in OS pagecache, effectively storing everything twice.
Furthermore, we are building on top of the JVM, and anyone who has spent any time with Java memory usage knows two things:
Download
@apachekafka
1. The memory overhead of objects is very high, often doubling the size of the data stored (or worse).
2. Java garbage collection becomes increasingly fiddly and slow as the in-heap data increases.
As a result of these factors using the filesystem and relying on pagecache is superior to maintaining an in-memory cache or other structurewe at least
double the available cache by having automatic access to all free memory, and likely double again by storing a compact byte structure rather than individual
objects. Doing so will result in a cache of up to 28-30GB on a 32GB machine without GC penalties. Furthermore, this cache will stay warm even if the service is
restarted, whereas the in-process cache will need to be rebuilt in memory (which for a 10GB cache may take 10 minutes) or else it will need to start with a
completely cold cache (which likely means terrible initial performance). This also greatly simplifies the code as all logic for maintaining coherency between
the cache and filesystem is now in the OS, which tends to do so more efficiently and more correctly than one-off in-process attempts. If your disk usage
favors linear reads then read-ahead is effectively pre-populating this cache with useful data on each disk read.
This suggests a design which is very simple: rather than maintain as much as possible in-memory and flush it all out to the filesystem in a panic when we run
out of space, we invert that. All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means
that it is transferred into the kernel's pagecache.
This style of pagecache-centric design is described in an article on the design of Varnish here (along with a healthy dose of arrogance).
4.3 Efficiency
We have put significant effort into efficiency. One of our primary use cases is handling web activity data, which is very high volume: each page view may
generate dozens of writes. Furthermore, we assume each message published is read by at least one consumer (often many), hence we strive to make
consumption as cheap as possible.
We have also found, from experience building and running a number of similar systems, that efficiency is a key to effective multi-tenant operations. If the
downstream infrastructure service can easily become a bottleneck due to a small bump in usage by the application, such small changes will often create
problems. By being very fast we help ensure that the application will tip-over under load before the infrastructure. This is particularly important when trying to
run a centralized service that supports dozens or hundreds of applications on a centralized cluster as changes in usage patterns are a near-daily occurrence.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Page 46 of 105
We discussed disk efficiency in the previous section. Once poor disk access patterns have been eliminated, there are two common causes of inefficiency in
this type of system: too many small I/O operations, and excessive byte copying.
The small I/O problem happens both between the client and the server and in the server's own persistent operations.
To avoid this, our protocol is built around a "message set" abstraction that naturally groups messages together. This allows network requests to group
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
messages together and amortize the overhead of the network roundtrip rather than sending a single message at a time. The server in turn appends chunks of
messages to its log in one go, and the consumer fetches large linear chunks at a time.
This simple optimization produces orders of magnitude speed up. Batching leads to larger network packets, larger sequential disk operations, contiguous
memory blocks, and so on, all of which allows Kafka to turn a bursty stream of random message writes into linear writes that flow to the consumers.
The other inefficiency is in byte copying. At low message rates this is not an issue, but under load the impact is significant. To avoid this we employ a
standardized binary message format that is shared by the producer, the broker, and the consumer (so data chunks can be transferred without modification
between them).
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
The message log maintained by the broker is itself just a directory of files, each populated by a sequence of message sets that have been written to disk in
the same format used by the producer and consumer. Maintaining this common format allows optimization of the most important operation: network transfer
of persistent log chunks. Modern unix operating systems offer a highly optimized code path for transferring data out of pagecache to a socket; in Linux this is
done with the sendfile system call.
To understand the impact of sendfile, it is important to understand the common data path for transfer of data from file to socket:
1. The operating system reads data from the disk into pagecache in kernel space
2. The application reads the data from kernel space into a user-space buffer
3. The application writes the data back into kernel space into a socket buffer
Download
@apachekafka
4. The operating system copies the data from the socket buffer to the NIC buffer where it is sent over the network
This is clearly inefficient, there are four copies and two system calls. Using sendfile, this re-copying is avoided by allowing the OS to send the data from
pagecache to the network directly. So in this optimized path, only the final copy to the NIC buffer is needed.
We expect a common use case to be multiple consumers on a topic. Using the zero-copy optimization above, data is copied into pagecache exactly once and
reused on each consumption instead of being stored in memory and copied out to kernel space every time it is read. This allows messages to be consumed
at a rate that approaches the limit of the network connection.
This combination of pagecache and sendfile means that on a Kafka cluster where the consumers are mostly caught up you will see no read activity on the
disks whatsoever as they will be serving data entirely from cache.
For more background on the sendfile and zero-copy support in Java, see this article.
Asynchronous send
Batching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger
batches in a single request. The batching can be configured to accumulate no more than a fixed number of messages and to wait no longer than some fixed
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Kafka Streams
Page 47 of 105
latency bound (say 64k or 10 ms). This allows the accumulation of more bytes to send, and few larger I/O operations on the servers. This buffering is
configurable and gives a mechanism to trade off a small amount of additional latency for better throughput.
Details on configuration and the api for the producer can be found elsewhere in the documentation.
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
An initial question we considered is whether consumers should pull data from brokers or brokers should push data to the consumer. In this respect Kafka
follows a more traditional design, shared by most messaging systems, where data is pushed to the broker from the producer and pulled from the broker by
the consumer. Some logging-centric systems, such as Scribe and Apache Flume, follow a very different push-based path where data is pushed downstream.
There are pros and cons to both approaches. However, a push-based system has difficulty dealing with diverse consumers as the broker controls the rate at
EVENTS
CONTACT US
APACHE
which data is transferred. The goal is generally for the consumer to be able to consume at the maximum possible rate; unfortunately, in a push system this
means the consumer tends to be overwhelmed when its rate of consumption falls below the rate of production (a denial of service attack, in essence). A pullbased system has the nicer property that the consumer simply falls behind and catches up when it can. This can be mitigated with some kind of backoff
protocol by which the consumer can indicate it is overwhelmed, but getting the rate of transfer to fully utilize (but never over-utilize) the consumer is trickier
than it seems. Previous attempts at building systems in this fashion led us to go with a more traditional pull model.
Download
Another advantage of a pull-based system is that it lends itself to aggressive batching of data sent to the consumer. A push-based system must choose to
either send a request immediately or accumulate more data and then send it later without knowledge of whether the downstream consumer will be able to
immediately process it. If tuned for low latency, this will result in sending a single message at a time only for the transfer to end up being buffered anyway,
@apachekafka
which is wasteful. A pull-based design fixes this as the consumer always pulls all available messages after its current position in the log (or up to some
configurable max size). So one gets optimal batching without introducing unnecessary latency.
The deficiency of a naive pull-based system is that if the broker has no data the consumer may end up polling in a tight loop, effectively busy-waiting for data
to arrive. To avoid this we have parameters in our pull request that allow the consumer request to block in a "long poll" waiting until data arrives (and
optionally waiting until a given number of bytes is available to ensure large transfer sizes).
You could imagine other possible designs which would be only pull, end-to-end. The producer would locally write to a local log, and brokers would pull from
that with consumers pulling from them. A similar type of "store-and-forward" producer is often proposed. This is intriguing but we felt not very suitable for our
target use cases which have thousands of producers. Our experience running persistent data systems at scale led us to feel that involving thousands of disks
in the system across many applications would not actually make things more reliable and would be a nightmare to operate. And in practice we have found
that we can run a pipeline with strong SLAs at large scale without a need for producer persistence.
Consumer Position
Keeping track of what has been consumed is, surprisingly, one of the key performance points of a messaging system.
Most messaging systems keep metadata about what messages have been consumed on the broker. That is, as a message is handed out to a consumer, the
broker either records that fact locally immediately or it may wait for acknowledgement from the consumer. This is a fairly intuitive choice, and indeed for a
single machine server it is not clear where else this state could go. Since the data structures used for storage in many messaging systems scale poorly, this is
also a pragmatic choice--since the broker knows what is consumed it can immediately delete it, keeping the data size small.
What is perhaps not obvious is that getting the broker and consumer to come into agreement about what has been consumed is not a trivial problem. If the
broker records a message as consumed immediately every time it is handed out over the network, then if the consumer fails to process the message (say
because it crashes or the request times out or whatever) that message will be lost. To solve this problem, many messaging systems add an
acknowledgement feature which means that messages are only marked as sent not consumed when they are sent; the broker waits for a specific
acknowledgement from the consumer to record the message as consumed. This strategy fixes the problem of losing messages, but creates new problems.
First of all, if the consumer processes the message but fails before it can send an acknowledgement then the message will be consumed twice. The second
problem is around performance, now the broker must keep multiple states about every single message (first to lock it so it is not given out a second time, and
then to mark it as permanently consumed so that it can be removed). Tricky problems must be dealt with, like what to do with messages that are sent but
never acknowledged.
Kafka handles this differently. Our topic is divided into a set of totally ordered partitions, each of which is consumed by exactly one consumer within each
subscribing consumer group at any given time. This means that the position of a consumer in each partition is just a single integer, the offset of the next
message to consume. This makes the state about what has been consumed very small, just one number for each partition. This state can be periodically
checkpointed. This makes the equivalent of message acknowledgements very cheap.
There is a side benefit of this decision. A consumer can deliberately rewind back to an old offset and re-consume data. This violates the common contract of
a queue, but turns out to be an essential feature for many consumers. For example, if the consumer code has a bug and is discovered after some messages
are consumed, the consumer can re-consume those messages once the bug is fixed.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 48 of 105
Offline Data Load
Scalable persistence allows for the possibility of consumers that only periodically consume such as batch data loads that periodically bulk-load data into an
offline system such as Hadoop or a relational data warehouse.
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
In the case of Hadoop we parallelize the data load by splitting the load over individual map tasks, one for each node/topic/partition combination, allowing full
parallelism in the loading. Hadoop provides the task management, and tasks which fail can restart without danger of duplicate datathey simply restart from
their original position.
Download
Kafka's semantics are straight-forward. When publishing a message we have a notion of the message being "committed" to the log. Once a published
message is committed it will not be lost as long as one broker that replicates the partition to which this message was written remains "alive". The definition of
@apachekafka
alive as well as a description of which types of failures we attempt to handle will be described in more detail in the next section. For now let's assume a
perfect, lossless broker and try to understand the guarantees to the producer and consumer. If a producer attempts to publish a message and experiences a
network error it cannot be sure if this error happened before or after the message was committed. This is similar to the semantics of inserting into a database
table with an autogenerated key.
These are not the strongest possible semantics for publishers. Although we cannot be sure of what happened in the case of a network error, it is possible to
allow the producer to generate a sort of "primary key" that makes retrying the produce request idempotent. This feature is not trivial for a replicated system
because of course it must work even (or especially) in the case of a server failure. With this feature it would suffice for the producer to retry until it receives
acknowledgement of a successfully committed message at which point we would guarantee the message had been published exactly once. We hope to add
this in a future Kafka version.
Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires. If the
producer specifies that it wants to wait on the message being committed this can take on the order of 10 ms. However the producer can also specify that it
wants to perform the send completely asynchronously or that it wants to wait only until the leader (but not necessarily the followers) have the message.
Now let's describe the semantics from the point-of-view of the consumer. All replicas have the exact same log with the same offsets. The consumer controls
its position in this log. If the consumer never crashed it could just store this position in memory, but if the consumer fails and we want this topic partition to
be taken over by another process the new process will need to choose an appropriate position from which to start processing. Let's say the consumer reads
some messages -- it has several options for processing the messages and updating its position.
1. It can read the messages, then save its position in the log, and finally process the messages. In this case there is a possibility that the consumer
process crashes after saving its position but before saving the output of its message processing. In this case the process that took over processing
would start at the saved position even though a few messages prior to that position had not been processed. This corresponds to "at-most-once"
semantics as in the case of a consumer failure messages may not be processed.
2. It can read the messages, process the messages, and finally save its position. In this case there is a possibility that the consumer process crashes
after processing messages but before saving its position. In this case when the new process takes over the first few messages it receives will already
have been processed. This corresponds to the "at-least-once" semantics in the case of consumer failure. In many cases messages have a primary key
and so the updates are idempotent (receiving the same message twice just overwrites a record with another copy of itself).
3. So what about exactly once semantics (i.e. the thing you actually want)? The limitation here is not actually a feature of the messaging system but rather
the need to co-ordinate the consumer's position with what is actually stored as output. The classic way of achieving this would be to introduce a twophase commit between the storage for the consumer position and the storage of the consumers output. But this can be handled more simply and
generally by simply letting the consumer store its offset in the same place as its output. This is better because many of the output systems a consumer
might want to write to will not support a two-phase commit. As an example of this, our Hadoop ETL that populates data in HDFS stores its offsets in
HDFS with the data it reads so that it is guaranteed that either data and offsets are both updated or neither is. We follow similar patterns for many other
data systems which require these stronger semantics and for which the messages do not have a primary key to allow for deduplication.
So effectively Kafka guarantees at-least-once delivery by default and allows the user to implement at most once delivery by disabling retries on the producer
and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka
provides the offset which makes implementing this straight-forward.
4.7 Replication
Kafka replicates the log for each topic's partitions across a configurable number of servers (you can set this replication factor on a topic-by-topic basis). This
allows automatic failover to these replicas when a server in the cluster fails so messages remain available in the presence of failures.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 49 of 105
Other messaging systems provide some replication-related features, but, in our (totally biased) opinion, this appears to be a tacked-on thing, not heavily used,
and with large downsides: slaves are inactive, throughput is heavily impacted, it requires fiddly manual configuration, etc. Kafka is meant to be used with
Operations
replication by defaultin fact we implement un-replicated topics as replicated topics where the replication factor is one.
Security
The unit of replication is the topic partition. Under non-failure conditions, each partition in Kafka has a single leader and zero or more followers. The total
Kafka Connect
number of replicas including the leader constitute the replication factor. All reads and writes go to the leader of the partition. Typically, there are many more
partitions than brokers and the leaders are evenly distributed among brokers. The logs on the followers are identical to the leader's logall have the same
Kafka Streams
offsets and messages in the same order (though, of course, at any given time the leader may have a few as-yet unreplicated messages at the end of its log).
PERFORMANCE
Followers consume messages from the leader just as a normal Kafka consumer would and apply them to their own log. Having the followers pull from the
leader has the nice property of allowing the follower to naturally batch together log entries they are applying to their log.
POWERED BY
As with most distributed systems automatically handling failures requires having a precise definition of what it means for a node to be "alive". For Kafka node
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Download
A message is considered "committed" when all in sync replicas for that partition have applied it to their log. Only committed messages are ever given out to
the consumer. This means that the consumer need not worry about potentially seeing a message that could be lost if the leader fails. Producers, on the other
@apachekafka
hand, have the option of either waiting for the message to be committed or not, depending on their preference for tradeoff between latency and durability.
This preference is controlled by the acks setting that the producer uses.
The guarantee that Kafka offers is that a committed message will not be lost, as long as there is at least one in sync replica alive, at all times.
Kafka will remain available in the presence of node failures after a short fail-over period, but may not remain available in the presence of network partitions.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 50 of 105
Kafka takes a slightly different approach to choosing its quorum set. Instead of majority vote, Kafka dynamically maintains a set of in-sync replicas (ISR) that
are caught-up to the leader. Only members of this set are eligible for election as leader. A write to a Kafka partition is not considered committed until
sync replicas have received the write. This ISR set is persisted to ZooKeeper whenever it changes. Because of this, any replica in the ISR is eligible to be
elected leader. This is an important factor for Kafka's usage model where there are many partitions and ensuring leadership balance is important. With this
Security
ISR model and f+1 replicas, a Kafka topic can tolerate f failures without losing committed messages.
Kafka Connect
Kafka Streams
For most use cases we hope to handle, we think this tradeoff is a reasonable one. In practice, to tolerate f failures, both the majority vote and the ISR
approach will wait for the same number of replicas to acknowledge before committing a message (e.g. to survive one failure a majority quorum needs three
replicas and one acknowledgement and the ISR approach requires two replicas and one acknowledgement). The ability to commit without the slowest servers
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
is an advantage of the majority vote approach. However, we think it is ameliorated by allowing the client to choose whether they block on the message
commit or not, and the additional throughput and disk space due to the lower required replication factor is worth it.
Another important design distinction is that Kafka does not require that crashed nodes recover with all their data intact. It is not uncommon for replication
algorithms in this space to depend on the existence of "stable storage" that cannot be lost in any failure-recovery scenario without potential consistency
violations. There are two primary problems with this assumption. First, disk errors are the most common problem we observe in real operation of persistent
data systems and they often do not leave data intact. Secondly, even if this were not a problem, we do not want to require the use of fsync on every write for
CLIENTS
EVENTS
CONTACT US
APACHE
our consistency guarantees as this can reduce performance by two to three orders of magnitude. Our protocol for allowing a replica to rejoin the ISR ensures
that before rejoining, it must fully re-sync again even if it lost unflushed data in its crash.
Download
However a practical system needs to do something reasonable when all the replicas die. If you are unlucky enough to have this occur, it is important to
consider what will happen. There are two behaviors that could be implemented:
@apachekafka
1. Wait for a replica in the ISR to come back to life and choose this replica as the leader (hopefully it still has all its data).
2. Choose the first replica (not necessarily in the ISR) that comes back to life as the leader.
This is a simple tradeoff between availability and consistency. If we wait for replicas in the ISR, then we will remain unavailable as long as those replicas are
down. If such replicas were destroyed or their data was lost, then we are permanently down. If, on the other hand, a non-in-sync replica comes back to life and
we allow it to become leader, then its log becomes the source of truth even though it is not guaranteed to have every committed message. By default, Kafka
chooses the second strategy and favor choosing a potentially inconsistent replica when all replicas in the ISR are dead. This behavior can be disabled using
configuration property unclean.leader.election.enable, to support use cases where downtime is preferable to inconsistency.
This dilemma is not specific to Kafka. It exists in any quorum-based scheme. For example in a majority voting scheme, if a majority of servers suffer a
permanent failure, then you must either choose to lose 100% of your data or violate consistency by taking what remains on an existing server as your new
source of truth.
Replica Management
The above discussion on replicated logs really covers only a single log, i.e. one topic partition. However a Kafka cluster will manage hundreds or thousands of
these partitions. We attempt to balance partitions within a cluster in a round-robin fashion to avoid clustering all partitions for high-volume topics on a small
number of nodes. Likewise we try to balance leadership so that each node is the leader for a proportional share of its partitions.
It is also important to optimize the leadership election process as that is the critical window of unavailability. A naive implementation of leader election would
end up running an election per partition for all partitions a node hosted when that node failed. Instead, we elect one of the brokers as the "controller". This
controller detects failures at the broker level and is responsible for changing the leader of all affected partitions in a failed broker. The result is that we are
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Page 51 of 105
able to batch together many of the required leadership change notifications which makes the election process far cheaper and faster for a large number of
partitions. If the controller fails, one of the surviving brokers will become the new controller.
Kafka Streams
operational maintenance. Let's dive into these use cases in more detail and then describe how compaction works.
PERFORMANCE
So far we have described only the simpler approach to data retention where old log data is discarded after a fixed period of time or when the log reaches
POWERED BY
PROJECT INFO
ECOSYSTEM
some predetermined size. This works well for temporal event data such as logging where each record stands alone. However an important class of data
streams are the log of changes to keyed, mutable data (for example, the changes to a database table).
Let's discuss a concrete example of such a stream. Say we have a topic containing user email addresses; every time a user updates their email address we
send a message to this topic using their user id as the primary key. Now say we send the following messages over some time period for a user with id 123,
each message corresponding to a change in email address (messages for other ids are omitted):
CLIENTS
123 => bill@microsoft.com
EVENTS
.
CONTACT US
.
.
APACHE
Download
.
123 => bill@gmail.com
@apachekafka
Log compaction gives us a more granular retention mechanism so that we are guaranteed to retain at least the last update for each primary key (e.g.
bill@gmail.com ). By doing this we guarantee that the log contains a full snapshot of the final value for every key not just keys that changed recently.
This means downstream consumers can restore their own state off this topic without us having to retain a complete log of all changes.
Let's start by looking at a few use cases where this is useful, then we'll see how it can be used.
1. Database change subscription. It is often necessary to have a data set in multiple data systems, and often one of these systems is a database of some
kind (either a RDBMS or perhaps a new-fangled key-value store). For example you might have a database, a cache, a search cluster, and a Hadoop
cluster. Each change to the database will need to be reflected in the cache, the search cluster, and eventually in Hadoop. In the case that one is only
handling the real-time updates you only need recent log. But if you want to be able to reload the cache or restore a failed search node you may need a
complete data set.
2. Event sourcing. This is a style of application design which co-locates query processing with application design and uses a log of changes as the
primary store for the application.
3. Journaling for high-availability. A process that does local computation can be made fault-tolerant by logging out changes that it makes to its local state
so another process can reload these changes and carry on if it should fail. A concrete example of this is handling counts, aggregations, and other
"group by"-like processing in a stream query system. Samza, a real-time stream-processing framework, uses this feature for exactly this purpose.
In each of these cases one needs primarily to handle the real-time feed of changes, but occasionally, when a machine crashes or data needs to be re-loaded
or re-processed, one needs to do a full load. Log compaction allows feeding both of these use cases off the same backing topic. This style of usage of a log is
described in more detail in this blog post.
The general idea is quite simple. If we had infinite log retention, and we logged each change in the above cases, then we would have captured the state of the
system at each time from when it first began. Using this complete log, we could restore to any point in time by replaying the first N records in the log. This
hypothetical complete log is not very practical for systems that update a single record many times as the log will grow without bound even for a stable
dataset. The simple log retention mechanism which throws away old updates will bound space but the log is no longer a way to restore the current statenow
restoring from the beginning of the log no longer recreates the current state as old updates may not be captured at all.
Log compaction is a mechanism to give finer-grained per-record retention, rather than the coarser-grained time-based retention. The idea is to selectively
remove records where we have a more recent update with the same primary key. This way the log is guaranteed to have at least the last state for each key.
This retention policy can be set per-topic, so a single cluster can have some topics where retention is enforced by size or time and other topics where
retention is enforced by compaction.
This functionality is inspired by one of LinkedIn's oldest and most successful pieces of infrastructurea database changelog caching service called
Unlike most log-structured storage systems Kafka is built for subscription and organizes data for fast linear reads and writes. Unlike Databus, Kafka acts as a
source-of-truth store so it is useful even in situations where the upstream data source would not otherwise be replayable.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 52 of 105
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
The head of the log is identical to a traditional Kafka log. It has dense, sequential offsets and retains all messages. Log compaction adds an option for
handling the tail of the log. The picture above shows a log with a compacted tail. Note that the messages in the tail of the log retain the original offset
POWERED BY
PROJECT INFO
ECOSYSTEM
assigned when they were first writtenthat never changes. Note also that all offsets remain valid positions in the log, even if the message with that offset has
been compacted away; in this case this position is indistinguishable from the next highest offset that does appear in the log. For example, in the picture above
the offsets 36, 37, and 38 are all equivalent positions and a read beginning at any of these offsets would return a message set beginning with 38.
Compaction also allows for deletes. A message with a key and a null payload will be treated as a delete from the log. This delete marker will cause any prior
CLIENTS
message with that key to be removed (as would any new message with that key), but delete markers are special in that they will themselves be cleaned out of
EVENTS
diagram.
CONTACT US
The compaction is done in the background by periodically recopying log segments. Cleaning does not block reads and can be throttled to use no more than a
APACHE
the log after a period of time to free up space. The point in time at which deletes are no longer retained is marked as the "delete retention point" in the above
configurable amount of I/O throughput to avoid impacting producers and consumers. The actual process of compacting a log segment looks something like
this:
Download
@apachekafka
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 53 of 105
4. The summary of the log head is essentially just a space-compact hash table. It uses exactly 24 bytes per entry. As a result with 8GB of cleaner buffer
one cleaner iteration can clean around 366GB of log head (assuming 1k messages).
Operations
Security
Kafka Connect
Kafka Streams
The log cleaner is enabled by default. This will start the pool of cleaner threads. To enable log cleaning on a particular topic you can add the log-specific
property
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
log.cleanup.policy=compact
This can be done either at topic creation time or using the alter topic command.
The log cleaner can be configured to retain a minimum amount of the uncompacted "head" of the log. This is enabled by setting the compaction time lag.
CLIENTS
EVENTS
CONTACT US
APACHE
log.cleaner.min.compaction.lag.ms
This can be used to prevent messages newer than a minimum message age from being subject to compaction. If not set, all log segments are eligible for
compaction except for the last segment, i.e. the one currently being written to. The active segment will not be compacted even if all of its messages are older
than the minimum compaction time lag.
Download
@apachekafka
4.9 Quotas
Starting in 0.9, the Kafka cluster has the ability to enforce quotas on produce and fetch requests. Quotas are basically byte-rate thresholds defined per group
of clients sharing a quota.
Client groups
The identity of Kafka clients is the user principal which represents an authenticated user in a secure cluster. In a cluster that supports unauthenticated clients,
user principal is a grouping of unauthenticated users chosen by the broker using a configurable PrincipalBuilder . Client-id is a logical grouping of
clients with a meaningful name chosen by the client application. The tuple (user, client-id) defines a secure logical group of clients that share both user
principal and client-id.
Quotas can be applied to (user, client-id), user or client-id groups. For a given connection, the most specific quota matching the connection is applied. All
connections of a quota group share the quota configured for the group. For example, if (user="test-user", client-id="test-client") has a produce quota of
10MB/sec, this is shared across all producer instances of user "test-user" with the client-id "test-client".
Quota Configuration
Quota configuration may be defined for (user, client-id), user and client-id groups. It is possible to override the default quota at any of the quota levels that
needs a higher (or even lower) quota. The mechanism is similar to the per-topic log config overrides. User and (user, client-id) quota overrides are written to
ZooKeeper under /config/users and client-id quota overrides are written under /config/clients. These overrides are read by all brokers and are effective
immediately. This lets us change quotas without having to do a rolling restart of the entire cluster. See here for details. Default quotas for each group may
also be updated dynamically using the same mechanism.
The order of precedence for quota configuration is:
1. /config/users/<user>/clients/<client-id>
2. /config/users/<user>/clients/<default>
3. /config/users/<user>
4. /config/users/<default>/clients/<client-id>
5. /config/users/<default>/clients/<default>
6. /config/users/<default>
7. /config/clients/<client-id>
8. /config/clients/<default>
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 54 of 105
Broker properties (quota.producer.default, quota.consumer.default) can also be used to set defaults for client-id groups. These properties are being
deprecated and will be removed in a later release. Default quotas for client-id can be set in Zookeeper similar to the other quota overrides and defaults.
Operations
Enforcement
Security
Kafka Connect
Kafka Streams
By default, each unique client group receives a fixed quota in bytes/sec as configured by the cluster. This quota is defined on a per-broker basis. Each client
can publish/fetch a maximum of X bytes/sec per broker before it gets throttled. We decided that defining these quotas per broker is much better than having
a fixed cluster wide bandwidth per client because that would require a mechanism to share client quota usage among all the brokers. This can be harder to
PERFORMANCE
POWERED BY
How does a broker react when it detects a quota violation? In our solution, the broker does not return an error rather it attempts to slow down a client
PROJECT INFO
ECOSYSTEM
CLIENTS
exceeding its quota. It computes the amount of delay needed to bring a guilty client under its quota and delays the response for that time. This approach
keeps the quota violation transparent to clients (outside of client-side metrics). This also keeps them from having to implement any special backoff and retry
behavior which can get tricky. In fact, bad client behavior (retry without backoff) can exacerbate the very problem quotas are trying to solve.
Client byte rate is measured over multiple small windows (e.g. 30 windows of 1 second each) in order to detect and correct quota violations quickly. Typically,
having large measurement windows (for e.g. 10 windows of 30 seconds each) leads to large bursts of traffic followed by long delays which is not great in
EVENTS
CONTACT US
5. IMPLEMENTATION
APACHE
Download
@apachekafka
Producer APIs
The Producer API that wraps the 2 low-level producers - kafka.producer.SyncProducer and kafka.producer.async.AsyncProducer
class Producer {
/* Sends the data, partitioned by key to the topic using either the */
/* synchronous or the asynchronous producer */
public void send(kafka.javaapi.producer.ProducerData<K,V> producerData);
/* Sends a list of data, partitioned by key to the topic using either */
/* the synchronous or the asynchronous producer */
public void send(java.util.List<kafka.javaapi.producer.ProducerData<K,V>> producerData);
/* Closes the producer and cleans up */
public void close();
}
The goal is to expose all the producer functionality through a single API to the client. The Kafka producer
can handle queueing/buffering of multiple producer requests and asynchronous dispatch of the batched data:
kafka.producer.Producer provides the ability to batch multiple produce requests ( producer.type=async ), before serializing and dispatching
them to the appropriate kafka broker partition. The size of the batch can be controlled by a few config parameters. As events enter a queue, they are
buffered in a queue, until either queue.time or batch.size is reached. A background thread
( kafka.producer.async.ProducerSendThread ) dequeues the batch of data and lets the kafka.producer.EventHandler serialize and
send the data to the appropriate kafka broker partition. A custom event handler can be plugged in through the event.handler config parameter. At
various stages of this producer queue pipeline, it is helpful to be able to inject callbacks, either for plugging in custom logging/tracing code or custom
monitoring logic. This is possible by implementing the kafka.producer.async.CallbackHandler interface and setting callback.handler
config parameter to that class.
handles the serialization of data through a user-specified Encoder :
interface Encoder<T> {
public Message toMessage(T data);
}
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 55 of 105
Implementation
interface Partitioner<T> {
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
The partition API uses the key and the number of available broker partitions to return a partition id. This id is used as an index into a sorted list of
broker_ids and partitions to pick a broker partition for the producer request. The default partitioning strategy is hash(key)%numPartitions
key is null, then a random broker partition is picked. A custom partitioning strategy can also be plugged in using the partitioner.class
parameter.
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
Consumer APIs
We have 2 levels of consumer APIs. The low-level "simple" API maintains a connection to a single broker and has a close correspondence to the network
requests sent to the server. This API is completely stateless, with the offset being passed in on every request, allowing the user to maintain this metadata
however they choose.
The high-level API hides the details of brokers from the consumer and allows consuming off the cluster of machines without concern for the underlying
topology. It also maintains the state of what has been consumed. The high-level API also provides the ability to subscribe to topics that match a filter
expression (i.e., either a whitelist or a blacklist regular expression).
APACHE
Low-level API
Download
class SimpleConsumer {
/* Send fetch request to a broker and get back a set of messages. */
@apachekafka
*/
public long[] getOffsetsBefore(String topic, int partition, long time, int maxNumOffsets);
}
The low-level API is used to implement the high-level API as well as being used directly for some of our offline consumers which have particular requirements
around maintaining state.
High-level API
*/
public Map<String,List<KafkaStream>> createMessageStreams(Map<String,Int> topicCountMap);
/**
* You can also obtain a list of KafkaStreams, that iterate over messages
* from topics that match a TopicFilter. (A TopicFilter encapsulates a
* whitelist or a blacklist which is a standard Java regex.)
*/
public List<KafkaStream> createMessageStreamsByFilter(
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 56 of 105
TopicFilter topicFilter, int numStreams);
Implementation
Operations
public commitOffsets()
Security
Kafka Connect
public shutdown()
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
This API is centered around iterators, implemented by the KafkaStream class. Each KafkaStream represents the stream of messages from one or more
partitions on one or more servers. Each stream is used for single threaded processing, so the client can provide the number of desired streams in the create
call. Thus a stream may represent the merging of multiple server partitions (to correspond to the number of processing threads), but each partition only goes
ECOSYSTEM
to one stream.
CLIENTS
The createMessageStreams call registers the consumer for the topic, which results in rebalancing the consumer/broker assignment. The API encourages
EVENTS
CONTACT US
APACHE
creating many topic streams in a single call in order to minimize this rebalancing. The createMessageStreamsByFilter call (additionally) registers watchers to
discover new topics that match its filter. Note that each stream that createMessageStreamsByFilter returns may iterate over messages from multiple topics
(i.e., if multiple topics are allowed by the filter).
Download
@apachekafka
MessageSet interface a writeTo method. This allows the file-backed message set to use the more efficient transferTo implementation instead
of an in-process buffered write. The threading model is a single acceptor thread and N processor threads which handle a fixed number of connections each.
This design has been pretty thoroughly tested elsewhere and found to be simple to implement and fast. The protocol is kept quite simple to allow for future
implementation of clients in other languages.
5.3 Messages
Messages consist of a fixed-size header, a variable length opaque key byte array and a variable length opaque value byte array. The header contains the
following fields:
A CRC32 checksum to detect corruption or truncation.
A format version.
An attributes identifier
A timestamp
Leaving the key and value opaque is the right decision: there is a great deal of progress being made on serialization libraries right now, and any particular
choice is unlikely to be right for all uses. Needless to say a particular application using Kafka would likely mandate a particular serialization type as part of its
usage. The MessageSet interface is simply an iterator over messages with specialized methods for bulk reading and writing to an NIO Channel
0 : no compression
1 : gzip
2 : snappy
*
*
*
*
*
3 : lz4
bit 3 : Timestamp type
0 : create time
1 : log append time
bit 4 ~ 7 : reserved
5.5 Log
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 57 of 105
A log for a topic named "my_topic" with two partitions consists of two directories (namely my_topic_0 and my_topic_1 ) populated with data files
containing the messages for that topic. The format of the log files is a sequence of "log entries""; each log entry is a 4 byte integer N storing the message
length which is followed by the N message bytes. Each message is uniquely identified by a 64-bit integer offset giving the byte position of the start of this
message in the stream of all messages ever sent to that topic on that partition. The on-disk format of each message is given below. Each log file is named
Security
with the offset of the first message it contains. So the first file created will be 00000000000.kafka, and each additional file will have an integer name roughly
Kafka Connect
bytes from the previous file where S is the max log file size given in the configuration.
Kafka Streams
The exact binary format for messages is versioned and maintained as a standard interface so message sets can be transferred between producer, broker, and
PERFORMANCE
POWERED BY
PROJECT INFO
offset
: 8 bytes
ECOSYSTEM
: 4 bytes
CLIENTS
magic value
: 1 byte
attributes
: 1 byte
timestamp
CONTACT US
key length
: 4 bytes
key
: K bytes
APACHE
value length
: 4 bytes
value
: V bytes
EVENTS
Download
The use of the message offset as the message id is unusual. Our original idea was to use a GUID generated by the producer, and maintain a mapping from
@apachekafka
GUID to offset on each broker. But since a consumer must maintain an ID for each server, the global uniqueness of the GUID provides no value. Furthermore,
the complexity of maintaining the mapping from a random id to an offset requires a heavy weight index structure which must be synchronized with disk,
essentially requiring a full persistent random-access data structure. Thus to simplify the lookup structure we decided to use a simple per-partition atomic
counter which could be coupled with the partition id and node id to uniquely identify a message; this makes the lookup structure simpler, though multiple
seeks per consumer request are still likely. However once we settled on a counter, the jump to directly using the offset seemed naturalboth after all are
monotonically increasing integers unique to a partition. Since the offset is hidden from the consumer API this decision is ultimately an implementation detail
and we went with the more efficient approach.
Writes
The log allows serial appends which always go to the last file. This file is rolled over to a fresh file when it reaches a configurable size (say 1GB). The log
takes two configuration parameters: M, which gives the number of messages to write before forcing the OS to flush the file to disk, and S, which gives a
number of seconds after which a flush is forced. This gives a durability guarantee of losing at most M messages or S seconds of data in the event of a
system crash.
Reads
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 58 of 105
Reads are done by giving the 64-bit logical offset of a message and an S-byte max chunk size. This will return an iterator over the messages contained in the
S-byte buffer. S is intended to be larger than any single message, but in the event of an abnormally large message, the read can be retried multiple times, each
time doubling the buffer size, until the message is read successfully. A maximum message and buffer size can be specified to make the server reject
messages larger than some size, and to give a bound to the client on the maximum it needs to ever read to get a complete message. It is likely that the read
Security
buffer ends with a partial message, this is easily detected by the size delimiting.
Kafka Connect
Kafka Streams
The actual process of reading from an offset requires first locating the log segment file in which the data is stored, calculating the file-specific offset from the
global offset value, and then reading from that file offset. The search is done as a simple binary search variation against an in-memory range maintained for
each file.
PERFORMANCE
The log provides the capability of getting the most recently written message to allow clients to start subscribing as of "right now". This is also useful in the
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
case the consumer fails to consume its data within its SLA-specified number of days. In this case when the client attempts to consume a non-existent offset
it is given an OutOfRangeException and can either reset itself or fail as appropriate to the use case.
The following is the format of the results sent to the consumer.
: 4 bytes
error code
: 2 bytes
message 1
: x bytes
...
message n
Download
@apachekafka
: x bytes
: 4 bytes
error code
: 2 bytes
messageSetSend 1
...
messageSetSend n
Deletes
Data is deleted one log segment at a time. The log manager allows pluggable delete policies to choose which files are eligible for deletion. The current policy
deletes any log with a modification time of more than N days ago, though a policy which retained the last N GB could also be useful. To avoid locking reads
while still allowing deletes that modify the segment list we use a copy-on-write style segment list implementation that provides consistent views to allow a
binary search to proceed on an immutable static snapshot view of the log segments while deletes are progressing.
Guarantees
The log provides a configuration parameter M which controls the maximum number of messages that are written before forcing a flush to disk. On startup a
log recovery process is run that iterates over all messages in the newest log segment and verifies that each message entry is valid. A message entry is valid if
the sum of its size and offset are less than the length of the file AND the CRC32 of the message payload matches the CRC stored with the message. In the
event corruption is detected the log is truncated to the last valid offset.
Note that two kinds of corruption must be handled: truncation in which an unwritten block is lost due to a crash, and corruption in which a nonsense block is
ADDED to the file. The reason for this is that in general the OS makes no guarantee of the write order between the file inode and the actual block data so in
addition to losing written data the file can gain nonsense data if the inode is updated with a new size but a crash occurs before the block containing that data
is written. The CRC detects this corner case, and prevents it from corrupting the log (though the unwritten messages are, of course, lost).
5.6 Distribution
Consumer Offset Tracking
The high-level consumer tracks the maximum offset it has consumed in each partition and periodically commits its offset vector so that it can resume from
those offsets in the event of a restart. Kafka provides the option to store all the offsets for a given consumer group in a designated broker (for that group)
called the offset manager. i.e., any consumer instance in that consumer group should send its offset commits and fetches to that offset manager (broker).
The high-level consumer handles this automatically. If you use the simple consumer you will need to manage offsets manually. This is currently unsupported
in the Java simple consumer which can only commit or fetch offsets in ZooKeeper. If you use the Scala simple consumer you can discover the offset
manager and explicitly commit or fetch offsets to the offset manager. A consumer can look up its offset manager by issuing a GroupCoordinatorRequest to
any Kafka broker and reading the GroupCoordinatorResponse which will contain the offset manager. The consumer can then proceed to commit or fetch
offsets from the offsets manager broker. In case the offset manager moves, the consumer will need to rediscover the offset manager. If you wish to manage
your offsets manually, you can take a look at these code samples that explain how to issue OffsetCommitRequest and OffsetFetchRequest.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 59 of 105
When the offset manager receives an OffsetCommitRequest, it appends the request to a special compacted Kafka topic named __consumer_offsets
offset manager sends a successful offset commit response to the consumer only after all the replicas of the offsets topic receive the offsets. In case the
offsets fail to replicate within a configurable timeout, the offset commit will fail and the consumer may retry the commit after backing off. (This is done
automatically by the high-level consumer.) The brokers periodically compact the offsets topic since it only needs to maintain the most recent offset commit
Security
per partition. The offset manager also caches the offsets in an in-memory table in order to serve offset fetches quickly.
Kafka Connect
Kafka Streams
When the offset manager receives an offset fetch request, it simply returns the last committed offset vector from the offsets cache. In case the offset
manager was just started or if it just became the offset manager for a new set of consumer groups (by becoming a leader for a partition of the offsets topic),
it may need to load the offsets topic partition into the cache. In this case, the offset fetch will fail with an OffsetsLoadInProgress exception and the consumer
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
may retry the OffsetFetchRequest after backing off. (This is done automatically by the high-level consumer.)
Migrating offsets from ZooKeeper to Kafka
Kafka consumers in earlier releases store their offsets by default in ZooKeeper. It is possible to migrate these consumers to commit offsets into Kafka by
following these steps:
1. Set offsets.storage=kafka and dual.commit.enabled=true in your consumer config.
2. Do a rolling bounce of your consumers and then verify that your consumers are healthy.
EVENTS
CONTACT US
APACHE
ZooKeeper Directories
Download
The following gives the ZooKeeper structures and algorithms used for co-ordination between consumers and brokers.
@apachekafka
Notation
When an element in a path is denoted [xyz], that means that the value of xyz is not fixed and there is in fact a ZooKeeper znode for each possible value of xyz.
For example /topics/[topic] would be a directory named /topics containing a sub-directory for each topic name. Numerical ranges are also given such as
[0...5] to indicate the subdirectories 0, 1, 2, 3, 4. An arrow -> is used to indicate the contents of a znode. For example /hello -> world would indicate a
znode /hello containing the value "world".
This is a list of all present broker nodes, each of which provides a unique logical broker id which identifies it to consumers (which must be given as part of its
configuration). On startup, a broker node registers itself by creating a znode with the logical broker id under /brokers/ids. The purpose of the logical broker id
is to allow a broker to be moved to a different physical machine without affecting consumers. An attempt to register a broker id that is already in use (say
because two servers are configured with the same broker id) results in an error.
Since the broker registers itself in ZooKeeper using ephemeral znodes, this registration is dynamic and will disappear if the broker is shutdown or dies (thus
notifying consumers it is no longer available).
Each broker registers itself under the topics it maintains and stores the number of partitions for that topic.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 60 of 105
Consumer Id Registry
In addition to the group_id which is shared by all consumers in a group, each consumer is given a transient, unique consumer_id (of the form hostname:uuid)
for identification purposes. Consumer ids are registered in the following directory.
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Download
@apachekafka
Each of the consumers in the group registers under its group and creates a znode with its consumer_id. The value of the znode contains a map of <topic,
#streams>. This id is simply used to identify each of the consumers which is currently active within a group. This is an ephemeral node so it will disappear if
the consumer process dies.
Consumer Offsets
Consumers track the maximum offset they have consumed in each partition. This value is stored in a ZooKeeper directory if
offsets.storage=zookeeper .
Cluster Id
The cluster id is a unique and immutable identifier assigned to a Kafka cluster. The cluster id can have a maximum of 22 characters and the allowed
characters are defined by the regular expression [a-zA-Z0-9_\-]+, which corresponds to the characters used by the URL-safe Base64 variant with no padding.
Conceptually, it is auto-generated when a cluster is started for the first time.
Implementation-wise, it is generated when a broker with version 0.10.1 or later is successfully started for the first time. The broker tries to get the cluster id
from the /cluster/id znode during startup. If the znode does not exist, the broker generates a new cluster id and creates the znode with this cluster id.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 61 of 105
design simplifies the implementation. Had we allowed a partition to be concurrently consumed by multiple consumers, there would be contention on the
partition and some kind of locking would be required. If there are more consumers than partitions, some consumers won't get any data at all. During
Operations
rebalancing, we try to assign partitions to consumers in such a way that reduces the number of broker nodes each consumer has to connect to.
Security
Kafka Connect
2.
PERFORMANCE
3.
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
4.
5.
sort CG
6.
7.
8.
9.
When rebalancing is triggered at one consumer, rebalancing should be triggered in other consumers within the same group about the same time.
6. OPERATIONS
Here is some information on actually running Kafka as a production system based on usage and experience at LinkedIn. Please send us any additional tips
you know of.
Download
This section will review the most common operations you will perform on your Kafka cluster. All of the tools reviewed in this section are available under the
bin/ directory of the Kafka distribution and each tool will print details on all possible commandline options if it is run with no arguments.
The replication factor controls how many servers will replicate each message that is written. If you have a replication factor of 3 then up to 2 servers can fail
before you will lose access to your data. We recommend you use a replication factor of 2 or 3 so that you can transparently bounce machines without
interrupting data consumption.
The partition count controls how many logs the topic will be sharded into. There are several impacts of the partition count. First each partition must fit entirely
on a single server. So if you have 20 partitions the full data set (and read and write load) will be handled by no more than 20 servers (no counting replicas).
Finally the partition count impacts the maximum parallelism of your consumers. This is discussed in greater detail in the concepts section.
Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a
dash (-) and the partition id. Since a typical folder name can not be over 255 characters long, there will be a limitation on the length of topic names. We
assume the number of partitions will not ever be above 100,000. Therefore, topic names cannot be longer than 249 characters. This leaves just enough room
in the folder name for a dash and a potentially 5 digit long partition id.
The configurations added on the command line override the default settings the server has for things like the length of time data should be retained. The
complete set of per-topic configurations is documented here.
Modifying topics
You can change the configuration or partitioning of a topic using the same topic tool.
To add partitions you can do
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 62 of 105
Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn't change the partitioning of existing data so this may
disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitions then this partitioning will
Operations
potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.
Security
To add configs:
Kafka Connect
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
To remove a config:
EVENTS
> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name
CONTACT US
APACHE
Topic deletion option is disabled by default. To enable it set the server config
Download
@apachekafka
delete.topic.enable=true
Kafka does not currently support reducing the number of partitions for a topic.
Instructions for changing the replication factor of a topic can be found here.
Graceful shutdown
The Kafka cluster will automatically detect any broker shutdown or failure and elect new leaders for the partitions on that machine. This will occur whether a
server fails or it is brought down intentionally for maintenance or configuration changes. For the latter cases Kafka supports a more graceful mechanism for
stopping a server than just killing it. When a server is stopped gracefully it has two optimizations it will take advantage of:
1. It will sync all its logs to disk to avoid needing to do any log recovery when it restarts (i.e. validating the checksum for all messages in the tail of the
log). Log recovery takes time so this speeds up intentional restarts.
2. It will migrate any partitions the server is the leader for to other replicas prior to shutting down. This will make the leadership transfer faster and
minimize the time each partition is unavailable to a few milliseconds.
Syncing the logs will happen automatically whenever the server is stopped other than by a hard kill, but the controlled leadership migration requires using a
special setting:
controlled.shutdown.enable=true
Note that controlled shutdown will only succeed if all the partitions hosted on the broker have replicas (i.e. the replication factor is greater than 1
one of these replicas is alive). This is generally what you want since shutting down the last replica would make that topic partition unavailable.
Balancing leadership
Whenever a broker stops or crashes leadership for that broker's partitions transfers to other replicas. This means that by default when the broker is restarted
it will only be a follower for all its partitions, meaning it will not be used for client reads and writes.
To avoid this imbalance, Kafka has a notion of preferred replicas. If the list of replicas for a partition is 1,5,9 then node 1 is preferred as the leader to either
node 5 or 9 because it is earlier in the replica list. You can have the Kafka cluster try to restore leadership to the restored replicas by running the command:
Since running this command can be tedious you can also configure Kafka to do this automatically by setting the following configuration:
auto.leader.rebalance.enable=true
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 63 of 105
Balancing Replicas Across Racks
The rack awareness feature spreads replicas of the same partition across different racks. This extends the guarantees Kafka provides for broker-failure to
cover rack-failure, limiting the risk of data loss should all the brokers on a rack fail at once. The feature can also be applied to other broker groupings such as
Security
Kafka Connect
You can specify that a broker belongs to a particular rack by adding a property to the broker config:
Kafka Streams
broker.rack=my-rack-id
PERFORMANCE
POWERED BY
PROJECT INFO
When a topic is created, modified or replicas are redistributed, the rack constraint will be honoured, ensuring replicas span as many racks as they can (a
partition will span min(#racks, replication-factor) different racks).
ECOSYSTEM
The algorithm used to assign replicas to brokers ensures that the number of leaders per broker will be constant, regardless of how brokers are distributed
CLIENTS
EVENTS
However if racks are assigned different numbers of brokers, the assignment of replicas will not be even. Racks with fewer brokers will get more replicas,
CONTACT US
APACHE
meaning they will use more storage and put more resources into replication. Hence it is sensible to configure an equal number of brokers per rack.
Download
single cluster. Kafka comes with a tool for mirroring data between Kafka clusters. The tool consumes from a source cluster and produces to a destination
cluster. A common use case for this kind of mirroring is to provide a replica in another datacenter. This scenario will be discussed in more detail in the next
@apachekafka
section.
You can run many such mirroring processes to increase throughput and for fault-tolerance (if one process dies, the others will take overs the additional load).
Data will be read from topics in the source cluster and written to a topic with the same name in the destination cluster. In fact the mirror maker is little more
than a Kafka consumer and producer hooked together.
The source and destination clusters are completely independent entities: they can have different numbers of partitions and the offsets will not be the same.
For this reason the mirror cluster is not really intended as a fault-tolerance mechanism (as the consumer position will be different); for that we recommend
using normal in-cluster replication. The mirror maker process will, however, retain and use the message key for partitioning so order is preserved on a per-key
basis.
Here is an example showing how to mirror a single topic (named my-topic) from an input cluster:
> bin/kafka-mirror-maker.sh
--consumer.config consumer.properties
--producer.config producer.properties --whitelist my-topic
Note that we specify the list of topics with the --whitelist option. This option allows any regular expression using Java-style regular expressions
you could mirror two topics named A and B using --whitelist 'A|B' . Or you could mirror all topics using --whitelist '*' . Make sure to quote
any regular expression to ensure the shell doesn't try to expand it as a file path. For convenience we allow the use of ',' instead of '|' to specify a list of topics.
Sometimes it is easier to say what it is that you don't want. Instead of using --whitelist to say what you want to mirror you can use --blacklist
say what to exclude. This also takes a regular expression argument. However, --blacklist is not supported when the new consumer has been enabled
(i.e. when bootstrap.servers has been defined in the consumer configuration).
Combining mirroring with the configuration auto.create.topics.enable=true makes it possible to have a replica cluster that will automatically
create and replicate all data in a source cluster even as new topics are added.
Topic
Pid Offset
logSize
Lag
Owner
my-group
my-topic
test_jkreps-mn-13
my-group
my-topic
test_jkreps-mn-13
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 64 of 105
NOTE: Since 0.9.0.0, the kafka.tools.ConsumerOffsetChecker tool has been deprecated. You should use the kafka.admin.ConsumerGroupCommand (or the
bin/kafka-consumer-groups.sh script) to manage consumer groups, including consumers created with the new consumer API.
Operations
Security
Kafka Connect
Kafka Streams
With the ConsumerGroupCommand tool, we can list, describe, or delete consumer groups. Note that deletion is only available when the group metadata is
stored in ZooKeeper. When using the new consumer API (where the broker handles coordination of partition handling and rebalance), the group is deleted
when the last committed offset for that group expires. For example, to list all consumer groups across all topics:
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
To view offsets as in the previous example with the ConsumerOffsetChecker, we "describe" the consumer group like this:
EVENTS
> bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group
CONTACT US
APACHE
GROUP
TOPIC
PARTITION
CURRENT-OFFSET
LOG-END-OFFSET
LAG
test-consumer-group
test-foo
Download
@apachekafka
If you are using the old high-level consumer and storing the group metadata in ZooKeeper (i.e. offsets.storage=zookeeper ), pass --zookeeper
instead of bootstrap-server :
The partition reassignment tool can be used to move some topics off of the current set of brokers to the newly added brokers. This is typically useful while
expanding an existing cluster since it is easier to move entire topics to the new set of brokers, than moving one partition at a time. When used to do this, the
user should provide a list of topics that should be moved to the new set of brokers and a target list of new brokers. The tool then evenly distributes all
partitions for the given list of topics across the new set of brokers. During this move, the replication factor of the topic is kept constant. Effectively the
replicas for all partitions for the input list of topics are moved from the old set of brokers to the newly added brokers.
For instance, the following example will move all partitions for topics foo1,foo2 to the new set of brokers 5,6. At the end of this move, all partitions for topics
foo1 and foo2 will only exist on brokers 5,6.
Since the tool accepts the input list of topics as a json file, you first need to identify the topics you want to move and create the json file as follows:
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 65 of 105
"version":1
Implementation
Operations
Security
Once the json file is ready, use the partition reassignment tool to generate a candidate assignment:
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
{"topic":"foo1","partition":0,"replicas":[3,4]},
{"topic":"foo2","partition":2,"replicas":[1,2]},
ECOSYSTEM
{"topic":"foo2","partition":0,"replicas":[3,4]},
{"topic":"foo1","partition":1,"replicas":[2,3]},
CLIENTS
EVENTS
CONTACT US
APACHE
{"topic":"foo2","partition":1,"replicas":[2,3]}]
}
Proposed partition reassignment configuration
{"version":1,
"partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
{"topic":"foo1","partition":0,"replicas":[5,6]},
Download
{"topic":"foo2","partition":2,"replicas":[5,6]},
{"topic":"foo2","partition":0,"replicas":[5,6]},
@apachekafka
{"topic":"foo1","partition":1,"replicas":[5,6]},
{"topic":"foo2","partition":1,"replicas":[5,6]}]
}
The tool generates a candidate assignment that will move all partitions from topics foo1,foo2 to brokers 5,6. Note, however, that at this point, the partition
movement has not started, it merely tells you the current assignment and the proposed new assignment. The current assignment should be saved in case you
want to rollback to it. The new assignment should be saved in a json file (e.g. expand-cluster-reassignment.json) to be input to the tool with the --execute
option as follows:
Finally, the --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json
(used with the --execute option) should be used with the --verify option:
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 66 of 105
Reassignment of partition [foo1,1] is in progress
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
The partition reassignment tool can also be used to selectively move replicas of a partition to a specific set of brokers. When used in this manner, it is
assumed that the user knows the reassignment plan and does not require the tool to generate a candidate reassignment, effectively skipping the --generate
step and moving straight to the --execute step
For instance, the following example moves partition 0 of topic foo1 to brokers 5,6 and partition 1 of topic foo2 to brokers 2,3:
The first step is to hand craft the custom reassignment plan in a json file:
{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":
Then, use the json file with the --execute option to start the reassignment process:
Download
@apachekafka
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},
{"topic":"foo2","partition":1,"replicas":[3,4]}]
}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},
{"topic":"foo2","partition":1,"replicas":[2,3]}]
}
The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used
with the --execute option) should be used with the --verify option:
Decommissioning brokers
The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. As such, the admin
has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can
be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To
make this process effortless, we plan to add tooling support for decommissioning brokers in the future.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 67 of 105
> cat increase-replication-factor.json
Implementation
Operations
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}
Security
Kafka Connect
Then, use the json file with the --execute option to start the reassignment process:
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with
the --execute option) should be used with the --verify option:
Download
@apachekafka
You can also verify the increase in replication factor with the kafka-topics tool:
PartitionCount:1
Topic: foo
Partition: 0
ReplicationFactor:3
Leader: 5
Configs:
When you execute this script you will see the throttle engage:
Should you wish to alter the throttle, during a rebalance, say to increase the throughput so it completes quicker, you can do this by re-running the execute
command passing the same reassignment-json-file:
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 68 of 105
Once the rebalance completes the administrator can check the status of the rebalance using the --verify option. If the rebalance has completed, the throttle
will be removed via the --verify command. It is important that administrators remove the throttle in a timely manner once rebalancing completes by running
Operations
the command with the --verify option. Failure to do so could cause regular replication traffic to be throttled.
Security
When the --verify option is executed, and the reassignment has completed, the script will confirm that the throttle was removed:
Kafka Connect
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
The administrator can also validate the assigned configs using the kafka-configs.sh. There are two pairs of throttle configuration used to manage the
throttling process. The throttle value itself. This is configured, at a broker level, using the dynamic properties:
leader.replication.throttled.rate
follower.replication.throttled.rate
Download
leader.replication.throttled.replicas
follower.replication.throttled.replicas
@apachekafka
Which are configured per topic. All four config values are automatically assigned by kafka-reassign-partitions.sh (discussed below).
To view the throttle limit configuration:
This shows the throttle applied to both leader and follower side of the replication protocol. By default both sides are assigned the same throttled throughput
value.
To view the list of throttled replicas:
Here we see the leader throttle is applied to partition 1 on broker 102 and partition 0 on broker 101. Likewise the follower throttle is applied to partition 1 on
broker 101 and partition 0 on broker 102.
By default kafka-reassign-partitions.sh will apply the leader throttle to all replicas that exist before the rebalance, any one of which might be leader. It will
apply the follower throttle to all move destinations. So if there is a partition with replicas on brokers 101,102, being reassigned to 102,103, a leader throttle, for
that partition, would be applied to 101,102 and a follower throttle would be applied to 103 only.
If required, you can also use the --alter switch on kafka-configs.sh to alter the throttle configurations manually.
Safe usage of throttled replication
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 69 of 105
Where BytesInPerSec is the metric that monitors the write throughput of producers into each broker.
Operations
The administrator can monitor whether replication is making progress, during the rebalance, using the metric:
Security
Kafka Connect
Kafka Streams
PERFORMANCE
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)
The lag should constantly decrease during replication. If the metric does not decrease the administrator should increase the throttle throughput as described
POWERED BY
above.
PROJECT INFO
Setting quotas
ECOSYSTEM
Quotas overrides and defaults may be configured at (user, client-id), user or client-id levels as described here. By default, clients receive an unlimited quota. It
CLIENTS
is possible to set custom quotas for each (user, client-id), user or client-id group.
EVENTS
CONTACT US
> bin/kafka-configs.sh
APACHE
Download
@apachekafka
> bin/kafka-configs.sh
> bin/kafka-configs.sh
It is possible to set default quotas for each (user, client-id), user or client-id group by specifying --entity-default option instead of --entity-name.
Configure default client-id quota for user=userA:
> bin/kafka-configs.sh
> bin/kafka-configs.sh
> bin/kafka-configs.sh
> bin/kafka-configs.sh
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 70 of 105
> bin/kafka-configs.sh
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
> bin/kafka-configs.sh
If entity name is not specified, all entities of the specified type are described. For example, describe all users:
> bin/kafka-configs.sh
APACHE
> bin/kafka-configs.sh
It is possible to set default quotas that apply to all client-ids by setting these configs on the brokers. These properties are applied only if quota overrides or
defaults are not configured in Zookeeper. By default, each client-id receives an unlimited quota. The following sets the default quota per producer and
consumer client-id to 10MB/sec.
quota.producer.default=10485760
quota.consumer.default=10485760
Note that these properties are being deprecated and may be removed in a future release. Defaults configured using kafka-configs.sh take precedence over
these properties.
6.2 Datacenters
Some deployments will need to manage a data pipeline that spans multiple datacenters. Our recommended approach to this is to deploy a local Kafka cluster
in each datacenter with application instances in each datacenter interacting only with their local cluster and mirroring between clusters (see the
documentation on the mirror maker tool for how to do this).
This deployment pattern allows datacenters to act as independent entities and allows us to manage and tune inter-datacenter replication centrally. This
allows each facility to stand alone and operate even if the inter-datacenter links are unavailable: when this occurs the mirroring falls behind until the link is
restored at which time it catches up.
For applications that need a global view of all data you can use mirroring to provide clusters which have aggregate data mirrored from the local clusters in
datacenters. These aggregate clusters are used for reads by applications that require the full data set.
This is not the only possible deployment pattern. It is possible to read from or write to a remote Kafka cluster over the WAN, though obviously this will add
whatever latency is required to get the cluster.
Kafka naturally batches data in both the producer and consumer so it can achieve high-throughput even over a high-latency connection. To allow this though it
may be necessary to increase the TCP socket buffer sizes for the producer, consumer, and broker using the socket.send.buffer.bytes
socket.receive.buffer.bytes configurations. The appropriate way to set this is documented here.
It is generally not advisable to run a single Kafka cluster that spans multiple datacenters over a high-latency link. This will incur very high replication latency
both for Kafka writes and ZooKeeper writes, and neither Kafka nor ZooKeeper will remain available in all locations if the network between locations is
unavailable.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Page 71 of 105
sync vs async production
batch size (for async producers)
The most important consumer configuration is the fetch size.
All configurations are documented in the configuration section.
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
# Replication configurations
num.replica.fetchers=4
ECOSYSTEM
replica.fetch.max.bytes=1048576
CLIENTS
replica.high.watermark.checkpoint.interval.ms=5000
EVENTS
replica.fetch.wait.max.ms=500
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
CONTACT US
replica.lag.time.max.ms=10000
APACHE
controller.socket.timeout.ms=30000
controller.message.queue.size=10
Download
# Log configuration
num.partitions=8
@apachekafka
message.max.bytes=1000000
auto.create.topics.enable=true
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.hours=168
log.flush.interval.ms=10000
log.flush.interval.messages=20000
log.flush.scheduler.interval.ms=2000
log.roll.hours=168
log.retention.check.interval.ms=300000
log.segment.bytes=1073741824
# ZK configuration
zookeeper.connection.timeout.ms=6000
zookeeper.sync.time.ms=2000
# Socket server configuration
num.io.threads=8
num.network.threads=8
socket.request.max.bytes=104857600
socket.receive.buffer.bytes=1048576
socket.send.buffer.bytes=1048576
queued.max.requests=16
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100
Our client configuration varies a fair amount between different use cases.
For reference, here are the stats on one of LinkedIn's busiest clusters (at peak):
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 72 of 105
60 brokers
50k partitions (replication factor 2)
800k messages/sec in
300 MB/sec inbound, 1 GB/sec+ outbound
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
The tuning looks fairly aggressive, but all of the brokers in that cluster have a 90% GC pause time of about 21ms, and they're doing less than 1 young GC per
second.
EVENTS
CONTACT US
OS
APACHE
Kafka should run well on any unix system and has been tested on Linux and Solaris.
We have seen a few issues running on Windows and Windows is not currently a well supported platform though we would be happy to change that.
Download
@apachekafka
It is unlikely to require much OS-level tuning, but there are two potentially important OS-level configurations:
File descriptor limits: Kafka uses file descriptors for log segments and open connections. If a broker hosts many partitions, consider that the broker needs
at least (number_of_partitions)*(partition_size/segment_size) to track all log segments in addition to the number of connections the broker makes. We
recommend at least 100000 allowed file descriptors for the broker processes as a starting point.
Max socket buffer size: can be increased to enable high-performance data transfer between data centers as described here.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 73 of 105
Understanding Linux OS Flush Behavior
In Linux, data written to the filesystem is maintained in pagecache until it must be written out to disk (due to an application-level fsync or the OS's own flush
policy). The flushing of data is done by a set of background threads called pdflush (or in post 2.6.32 kernels "flusher threads").
Security
Kafka Connect
Pdflush has a configurable policy that controls how much dirty data can be maintained in cache and for how long before it must be written back to disk. This
policy is described here. When Pdflush cannot keep up with the rate of data being written it will eventually cause the writing process to block incurring latency
Kafka Streams
PERFORMANCE
POWERED BY
> cat /proc/meminfo
PROJECT INFO
ECOSYSTEM
The meaning of these values are described in the link above.
CLIENTS
Using pagecache has several advantages over an in-process cache for storing data that will be written out to disk:
EVENTS
The I/O scheduler will batch together consecutive small writes into bigger physical writes which improves throughput.
CONTACT US
APACHE
The I/O scheduler will attempt to re-sequence writes to minimize movement of the disk head which improves throughput.
It automatically uses all the free memory on the machine
Filesystem Selection
Download
@apachekafka
Kafka uses regular files on disk, and as such it has no hard dependency on a specific filesystem. The two filesystems which have the most usage, however,
are EXT4 and XFS. Historically, EXT4 has had more usage, but recent improvements to the XFS filesystem have shown it to have better performance
characteristics for Kafka's workload with no compromise in stability.
Comparison testing was performed on a cluster with significant message loads, using a variety of filesystem creation and mount options. The primary metric
in Kafka that was monitored was the "Request Local Time", indicating the amount of time append operations were taking. XFS resulted in much better local
times (160ms vs. 250ms+ for the best EXT4 configuration), as well as lower average wait times. The XFS performance also showed less variability in disk
performance.
General Filesystem Notes
For any filesystem used for data directories, on Linux systems, the following options are recommended to be used at mount time:
noatime: This option disables updating of a file's atime (last access time) attribute when the file is read. This can eliminate a significant number of
filesystem writes, especially in the case of bootstrapping consumers. Kafka does not rely on the atime attributes at all, so it is safe to disable this.
XFS Notes
The XFS filesystem has a significant amount of auto-tuning in place, so it does not require any change in the default settings, either at filesystem creation
time or at mount. The only tuning parameters worth considering are:
largeio: This affects the preferred I/O size reported by the stat call. While this can allow for higher performance on larger disk writes, in practice it had
minimal or no effect on performance.
nobarrier: For underlying devices that have battery-backed cache, this option can provide a little more performance by disabling periodic write flushes.
However, if the underlying device is well-behaved, it will report to the filesystem that it does not require flushes, and this option will have no effect.
EXT4 Notes
EXT4 is a serviceable choice of filesystem for the Kafka data directories, however getting the most performance out of it will require adjusting several mount
options. In addition, these options are generally unsafe in a failure scenario, and will result in much more data loss and corruption. For a single broker failure,
this is not much of a concern as the disk can be wiped and the replicas rebuilt from the cluster. In a multiple-failure scenario, such as a power outage, this can
mean underlying filesystem (and therefore data) corruption that is not easily recoverable. The following options can be adjusted:
data=writeback: Ext4 defaults to data=ordered which puts a strong order on some writes. Kafka does not require this ordering as it does very paranoid
data recovery on all unflushed log. This setting removes the ordering constraint and seems to significantly reduce latency.
Disabling journaling: Journaling is a tradeoff: it makes reboots faster after server crashes but it introduces a great deal of additional locking which adds
variance to write performance. Those who don't care about reboot time and want to reduce a major source of write latency spikes can turn off journaling
entirely.
commit=num_secs: This tunes the frequency with which ext4 commits to its metadata journal. Setting this to a lower value reduces the loss of unflushed
data during a crash. Setting this to a higher value will improve throughput.
nobh: This setting controls additional ordering guarantees when using data=writeback mode. This should be safe with Kafka as we do not depend on write
ordering and improves throughput and latency.
delalloc: Delayed allocation means that the filesystem avoid allocating any blocks until the physical write occurs. This allows ext4 to allocate a large extent
instead of smaller pages and helps ensure the data is written sequentially. This feature is great for throughput. It does seem to involve some locking in the
filesystem which adds a bit of latency variance.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 74 of 105
6.6 Monitoring
Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to
hook up to your monitoring system.
Security
Kafka Connect
Kafka Streams
PERFORMANCE
The easiest way to see the available metrics is to fire up jconsole and point it at a running kafka client or server; this will allow browsing all metrics with JMX.
We do graphing and alerting on the following metrics:
DESCRIPTION
POWERED BY
Message in rate
PROJECT INFO
ECOSYSTEM
Byte in rate
CLIENTS
EVENTS
Download
Request rate
kafka.server:type=BrokerTopicMetri
cs,name=MessagesInPerSec
kafka.server:type=BrokerTopicMetri
cs,name=BytesInPerSec
s,name=RequestsPerSec,request=
{Produce|FetchConsumer|FetchFoll
ower}
@apachekafka
NORMAL VALUE
kafka.network:type=RequestMetric
CONTACT US
APACHE
MBEAN NAME
kafka.server:type=BrokerTopicMetri
cs,name=BytesOutPerSec
kafka.log:type=LogFlushStats,name
=LogFlushRateAndTimeMs
kafka.server:type=ReplicaManager,
name=UnderReplicatedPartitions
kafka.controller:type=KafkaControll
ker
er,name=ActiveControllerCount
kafka.controller:type=ControllerStat
Leader election rate
s,name=LeaderElectionRateAndTim
eMs
Unclean leader election r
ate
Partition counts
kafka.controller:type=ControllerStat
s,name=UncleanLeaderElectionsPe
rSec
kafka.server:type=ReplicaManager,
name=PartitionCount
kafka.server:type=ReplicaManager,
name=LeaderCount
kafka.server:type=ReplicaManager,
name=IsrShrinksPerSec
kafka.server:type=ReplicaManager,
name=IsrExpandsPerSec
kafka.server:type=ReplicaFetcherM
anager,name=MaxLag,clientId=Repl
licas
ica
kafka.server:type=FetcherLagMetri
Lag in messages per foll
cs,name=ConsumerLag,clientId=
ower replica
([-.\w]+),topic=([-.\w]+),partition=([0
-9]+)
Requests waiting in the p
roducer purgatory
kafka.server:type=DelayedOperatio
nPurgatory,name=PurgatorySize,del
ayedOperation=Produce
kafka.server:type=DelayedOperatio
nPurgatory,name=PurgatorySize,del
ayedOperation=Fetch
kafka.network:type=RequestMetric
https://kafka.apache.org/documentation
s,name=TotalTimeMs,request={Pro
duce|FetchConsumer|FetchFollowe
r}
11/4/2016
Apache Kafka
Implementation
Page 75 of 105
Time the request waits in
kafka.network:type=RequestMetric
s,name=RequestQueueTimeMs,req
uest={Produce|FetchConsumer|Fet
Operations
chFollower}
Security
kafka.network:type=RequestMetric
Kafka Connect
Kafka Streams
s,name=LocalTimeMs,request={Pro
duce|FetchConsumer|FetchFollowe
r}
PERFORMANCE
POWERED BY
PROJECT INFO
kafka.network:type=RequestMetric
Time the request waits f
s,name=RemoteTimeMs,request=
or the follower
{Produce|FetchConsumer|FetchFoll
ower}
ECOSYSTEM
CLIENTS
kafka.network:type=RequestMetric
Time the request waits in
s,name=ResponseQueueTimeMs,re
quest={Produce|FetchConsumer|Fe
tchFollower}
EVENTS
CONTACT US
APACHE
kafka.network:type=RequestMetric
Time to send the respon
s,name=ResponseSendTimeMs,req
se
uest={Produce|FetchConsumer|Fet
chFollower}
Download
@apachekafka
kafka.consumer:type=ConsumerFet
cherManager,name=MaxLag,clientI
e producer by
d=([-.\w]+)
kafka.network:type=SocketServer,n
ame=NetworkProcessorAvgIdlePer
rs are idle
cent
kafka.server:type=KafkaRequestHa
ndlerPool,name=RequestHandlerAv
gIdlePercent
Two attributes. throttle-time indicates the amount of time in ms the client was thr
Quota metrics per (user,
kafka.server:type={Produce|Fetch},
user=([-.\w]+),client-id=([-.\w]+)
ottled. Ideally = 0. byte-rate indicates the data produce/consume rate of the client
in bytes/sec. For (user, client-id) quotas, both user and client-id are specified. If pe
r-client-id quota is applied to the client, user is not specified. If per-user quota is a
pplied, client-id is not specified.
METRIC/ATTRIBUTE
NAME
connection-close-rate
connection-creation-rat
e
network-io-rate
outgoing-byte-rate
DESCRIPTION
MBEAN NAME
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
rvers.
umer|connect]-metrics,client-id=([-.\w]+)
request-rate
request-size-avg
request-size-max
incoming-byte-rate
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 76 of 105
response-rate
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
Operations
Number of times the I/O layer checked for new I/O to perform p
kafka.[producer|consumer|connect]:type=[producer|cons
Security
er second.
umer|connect]-metrics,client-id=([-.\w]+)
Kafka Connect
The average length of time the I/O thread spent waiting for a so
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
select-rate
io-wait-time-ns-avg
Kafka Streams
PERFORMANCE
POWERED BY
io-wait-ratio
io-time-ns-avg
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
The average length of time for I/O per select call in nanosecond
kafka.[producer|consumer|connect]:type=[producer|cons
s.
umer|connect]-metrics,client-id=([-.\w]+)
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
io-ratio
connection-count
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
kafka.[producer|consumer|connect]:type=[producer|cons
umer|connect]-metrics,client-id=([-.\w]+)
The following metrics are available on producer/consumer/connector instances. For specific metrics, please see following sections.
Download
@apachekafka
METRIC/ATTRIBUTE
NAME
outgoing-byte-rate
request-rate
request-size-avg
request-size-max
incoming-byte-rate
DESCRIPTION
MBEAN NAME
kafka.producer:type=[consumer|producer|connect]-node-metrics,client-
id=([-.\w]+),node-id=([0-9]+)
kafka.producer:type=[consumer|producer|connect]-node-metrics,client-
for a node.
id=([-.\w]+),node-id=([0-9]+)
kafka.producer:type=[consumer|producer|connect]-node-metrics,client-
a node.
id=([-.\w]+),node-id=([0-9]+)
kafka.producer:type=[consumer|producer|connect]-node-metrics,client-
id=([-.\w]+),node-id=([0-9]+)
kafka.producer:type=[consumer|producer|connect]-node-metrics,client-
id=([-.\w]+),node-id=([0-9]+)
request-latency-avg
request-latency-max
response-rate
kafka.producer:type=[consumer|producer|connect]-node-metrics,clientid=([-.\w]+),node-id=([0-9]+)
kafka.producer:type=[consumer|producer|connect]-node-metrics,clientid=([-.\w]+),node-id=([0-9]+)
kafka.producer:type=[consumer|producer|connect]-node-metrics,clientid=([-.\w]+),node-id=([0-9]+)
Producer monitoring
The following metrics are available on producer instances.
METRIC/ATTRIBUTE
NAME
waiting-threads
buffer-total-bytes
buffer-available-bytes
DESCRIPTION
MBEAN NAME
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
The maximum amount of buffer memory the client can use (wheth
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
The total amount of buffer memory that is not being used (either u
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
bufferpool-wait-time
batch-size-avg
https://kafka.apache.org/documentation
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
11/4/2016
Apache Kafka
Implementation
Page 77 of 105
batch-size-max
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
Operations
compression-rate-avg
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
Security
Kafka Connect
record-queue-time-avg
Kafka Streams
PERFORMANCE
POWERED BY
record-queue-time-max
kafka.producer:type=producer-metrics,client-id=([-.\w]
ulator.
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
mulator.
+)
request-latency-avg
request-latency-max
record-send-rate
records-per-request-avg
record-retry-rate
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
PROJECT INFO
ECOSYSTEM
CLIENTS
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
EVENTS
CONTACT US
APACHE
Download
@apachekafka
record-error-rate
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
rrors.
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
record-size-max
record-size-avg
requests-in-flight
metadata-age
record-send-rate
byte-rate
compression-rate
record-retry-rate
record-error-rate
produce-throttle-time-ma
x
produce-throttle-time-av
g
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-metrics,client-id=([-.\w]
+)
kafka.producer:type=producer-topic-metrics,client-id=
([-.\w]+),topic=([-.\w]+)
kafka.producer:type=producer-topic-metrics,client-id=
([-.\w]+),topic=([-.\w]+)
kafka.producer:type=producer-topic-metrics,client-id=
([-.\w]+),topic=([-.\w]+)
kafka.producer:type=producer-topic-metrics,client-id=
c.
([-.\w]+),topic=([-.\w]+)
kafka.producer:type=producer-topic-metrics,client-id=
([-.\w]+),topic=([-.\w]+)
kafka.producer:type=producer-topic-metrics,client-id=
([-.\w]+)
kafka.producer:type=producer-topic-metrics,client-id=
([-.\w]+)
METRIC/ATTRIBUTE NAME
DESCRIPTION
commit-latency-avg
commit-latency-max
commit-rate
https://kafka.apache.org/documentation
MBEAN NAME
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
11/4/2016
Apache Kafka
Page 78 of 105
kafka.consumer:type=consumer-coordinator-metrics,client-i
Implementation
d=([-.\w]+)
Operations
kafka.consumer:type=consumer-coordinator-metrics,client-i
sumer
d=([-.\w]+)
heartbeat-response-time-m
kafka.consumer:type=consumer-coordinator-metrics,client-i
ax
at request
d=([-.\w]+)
heartbeat-rate
join-time-avg
join-time-max
join-rate
sync-time-avg
sync-time-max
sync-rate
assigned-partitions
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Download
@apachekafka
last-heartbeat-seconds-ago
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
d=([-.\w]+)
kafka.consumer:type=consumer-coordinator-metrics,client-i
eat
d=([-.\w]+)
METRIC/ATTRIBUTE
NAME
DESCRIPTION
fetch-size-avg
fetch-size-max
bytes-consumed-rate
records-per-request-avg
records-consumed-rate
fetch-latency-avg
fetch-latency-max
fetch-rate
records-lag-max
MBEAN NAME
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
on in this window
lient-id=([-.\w]+)
fetch-throttle-time-avg
fetch-throttle-time-max
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,c
lient-id=([-.\w]+)
METRIC/ATTRIBUTE
NAME
DESCRIPTION
MBEAN NAME
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 79 of 105
fetch-size-avg
Operations
kafka.consumer:type=consumer-fetch-manager-metrics,client-id
specific topic.
=([-.\w]+),topic=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,client-id
Security
a specific topic.
=([-.\w]+),topic=([-.\w]+)
Kafka Connect
kafka.consumer:type=consumer-fetch-manager-metrics,client-id
a specific topic.
=([-.\w]+),topic=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,client-id
ecific topic.
=([-.\w]+),topic=([-.\w]+)
kafka.consumer:type=consumer-fetch-manager-metrics,client-id
or a specific topic.
=([-.\w]+),topic=([-.\w]+)
fetch-size-max
bytes-consumed-rate
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
records-per-request-avg
records-consumed-rate
Others
We recommend monitoring GC time and other stats and various server stats such as CPU utilization, I/O service time, etc. On the client side, we recommend
monitoring the message/byte rate (global and per topic), request rate/size/time, and on the consumer side, max lag in messages among all partitions and min
fetch request rate. For a consumer to keep up, max lag needs to be less than a threshold and min fetch rate needs to be larger than 0.
CONTACT US
APACHE
Audit
The final alerting we do is on the correctness of the data delivery. We audit that every message that is sent is consumed by all consumers and measure the
Download
lag for this to occur. For important topics we alert if a certain completeness is not achieved in a certain time period. The details of this are discussed in
KAFKA-260.
@apachekafka
6.7 ZooKeeper
Stable version
The current stable branch is 3.4 and the latest release of that branch is 3.4.8, which is the one ZkClient 0.9 uses. ZkClient is the client layer Kafka uses to
interact with ZooKeeper.
Operationalizing ZooKeeper
Operationally, we do the following for a healthy ZooKeeper installation:
Redundancy in the physical/hardware/network layout: try not to put them all in the same rack, decent (but don't go nuts) hardware, try to keep redundant
power and network paths, etc. A typical ZooKeeper ensemble has 5 or 7 servers, which tolerates 2 and 3 servers down, respectively. If you have a small
deployment, then using 3 servers is acceptable, but keep in mind that you'll only be able to tolerate 1 server down in this case.
I/O segregation: if you do a lot of write type traffic you'll almost definitely want the transaction logs on a dedicated disk group. Writes to the transaction log
are synchronous (but batched for performance), and consequently, concurrent writes can significantly affect performance. ZooKeeper snapshots can be
one such a source of concurrent writes, and ideally should be written on a disk group separate from the transaction log. Snapshots are written to disk
asynchronously, so it is typically ok to share with the operating system and message log files. You can configure a server to use a separate disk group with
the dataLogDir parameter.
Application segregation: Unless you really understand the application patterns of other apps that you want to install on the same box, it can be a good idea
to run ZooKeeper in isolation (though this can be a balancing act with the capabilities of the hardware).
Use care with virtualization: It can work, depending on your cluster layout and read/write patterns and SLAs, but the tiny overheads introduced by the
virtualization layer can add up and throw off ZooKeeper, as it can be very time sensitive
ZooKeeper configuration: It's java, make sure you give it 'enough' heap space (We usually run them with 3-5G, but that's mostly due to the data set size we
have here). Unfortunately we don't have a good formula for it, but keep in mind that allowing for more ZooKeeper state means that snapshots can become
large, and large snapshots affect recovery time. In fact, if the snapshot becomes too large (a few gigabytes), then you may need to increase the initLimit
parameter to give enough time for servers to recover and join the ensemble.
Monitoring: Both JMX and the 4 letter words (4lw) commands are very useful, they do overlap in some cases (and in those cases we prefer the 4 letter
commands, they seem more predictable, or at the very least, they work better with the LI monitoring infrastructure)
Don't overbuild the cluster: large clusters, especially in a write heavy usage pattern, means a lot of intracluster communication (quorums on the writes and
subsequent cluster member updates), but don't underbuild it (and risk swamping the cluster). Having more servers adds to your read capacity.
Overall, we try to keep the ZooKeeper system as small as will handle the load (plus standard growth capacity planning) and as simple as possible. We try not
to do anything fancy with the configuration or application layout as compared to the official release as well as keep it as self contained as possible. For these
reasons, we tend to skip the OS packaged versions, since it has a tendency to try to put things in the OS standard hierarchy, which can be 'messy', for want of
a better way to word it.
7. SECURITY
7.1 Security Overview
In release 0.9.0.0, the Kafka community added a number of features that, used either separately or together, increases security in a Kafka cluster. These
features are considered to be of beta quality. The following security measures are currently supported:
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 80 of 105
1. Authentication of connections to brokers from clients (producers and consumers), other brokers and tools, using either SSL or SASL (Kerberos).
SASL/PLAIN can also be used from release 0.10.0.0 onwards.
2. Authentication of connections from brokers to ZooKeeper
3. Encryption of data transferred between brokers and clients, between brokers, or between brokers and tools using SSL (Note that there is a performance
Security
degradation when SSL is enabled, the magnitude of which depends on the CPU type and the JVM implementation.)
Kafka Connect
Kafka Streams
It's worth noting that security is optional - non-secured clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and nonPERFORMANCE
POWERED BY
encrypted clients. The guides below explain how to configure and use the security features in both clients and brokers.
PROJECT INFO
Apache Kafka allows clients to connect over SSL. By default, SSL is disabled but can be turned on as needed.
ECOSYSTEM
CLIENTS
1.
EVENTS
CONTACT US
APACHE
Download
@apachekafka
safely.
2. validity: the valid time of the certificate in days.
Note: By default the property ssl.endpoint.identification.algorithm is not defined, so hostname verification is not performed. In order to
enable hostname verification, set the following property:
ssl.endpoint.identification.algorithm=HTTPS
Once enabled, clients will verify the server's fully qualified domain name (FQDN) against one of the following two fields:
1. Common Name (CN)
2. Subject Alternative Name (SAN)
Both fields are valid, RFC-2818 recommends the use of SAN however. SAN is also more flexible, allowing for multiple DNS entries to be declared.
Another advantage is that the CN can be set to a more meaningful value for authorization purposes. To add a SAN field append the following argument
-ext SAN=DNS:{FQDN}
keytool -keystore server.keystore.jks -alias localhost -validity {validity} -genkey -ext SAN=DNS:{FQDN}
The following command can be run afterwards to verify the contents of the generated certificate:
2.
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 81 of 105
Implementation
The generated CA is simply a public-private key pair and certificate, and it is intended to sign other certificates.
Operations
The next step is to add the generated CA to the **clients' truststore** so that the clients can trust this CA:
Security
Kafka Connect
Kafka Streams
PERFORMANCE
Note: If you configure the Kafka brokers to require client authentication by setting ssl.client.auth to be "requested" or "required" on the Kafka brokers
config then you must provide a truststore for the Kafka brokers as well and it should have all the CA certificates that clients' keys were signed by.
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
In contrast to the keystore in step 1 that stores each machine's own identity, the truststore of a client stores all the certificates that the client should
trust. Importing a certificate into one's truststore also means trusting all certificates that are signed by that certificate. As the analogy above, trusting
EVENTS
the government (CA) also means trusting all passports (certificates) that it has issued. This attribute is called the chain of trust, and it is particularly
useful when deploying SSL on a large Kafka cluster. You can sign all certificates in the cluster with a single CA, and have all machines share the same
CONTACT US
truststore that trusts the CA. That way all machines can authenticate all other machines.
APACHE
3.
Download
@apachekafka
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateser
Finally, you need to import both the certificate of the CA and the signed certificate into the keystore:
#!/bin/bash
#Step 1
keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
#Step 3
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -pa
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 82 of 105
4.
Security
Kafka Connect
Kafka Streams
PERFORMANCE
listeners
If SSL is not enabled for inter-broker communication (see below for how to enable it), both PLAINTEXT and SSL ports will be necessary.
POWERED BY
listeners=PLAINTEXT://host.name:port,SSL://host.name:port
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=test1234
CONTACT US
ssl.key.password=test1234
APACHE
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=test1234
Download
@apachekafka
1. ssl.client.auth=none ("required" => client authentication is required, "requested" => client authentication is requested and client without certs can
still connect. The usage of "requested" is discouraged as it provides a false sense of security and misconfigured clients will still connect
successfully.)
2. ssl.cipher.suites (Optional). A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to
negotiate the security settings for a network connection using TLS or SSL network protocol. (Default is an empty list)
3. ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 (list out the SSL protocols that you are going to accept from clients. Do note that SSL is
deprecated in favor of TLS and using SSL in production is not recommended)
4. ssl.keystore.type=JKS
5. ssl.truststore.type=JKS
6. ssl.secure.random.implementation=SHA1PRNG
If you want to enable SSL for inter-broker communication, add the following to the broker properties file (it defaults to PLAINTEXT)
security.inter.broker.protocol=SSL
Due to import regulations in some countries, the Oracle implementation limits the strength of cryptographic algorithms available by default. If stronger
algorithms are needed (for example, AES with 256-bit keys), the JCE Unlimited Strength Jurisdiction Policy Files must be obtained and installed in the
JDK/JRE. See the JCA Providers Documentation for more information.
The JRE/JDK will have a default pseudo-random number generator (PRNG) that is used for cryptography operations, so it is not required to configure
the implementation used with the
ssl.secure.random.implementation
. However, there are performance issues with some implementations (notably, the default chosen on Linux systems,
NativePRNG
, utilizes a global lock). In cases where performance of SSL connections becomes an issue, consider explicitly setting the implementation to be used.
The
SHA1PRNG
implementation is non-blocking, and has shown very good performance characteristics under heavy load (50 MB/sec of produced messages, plus
replication traffic, per-broker).
Once you start the broker you should be able to see in the server.log
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 83 of 105
Implementation
Operations
Security
To check quickly if the server keystore and truststore are setup properly you can run the following command
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
-----BEGIN CERTIFICATE-----
ECOSYSTEM
CLIENTS
issuer=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=kafka/emailAddress=test@test.com
CONTACT US
APACHE
Download
If the certificate does not show up or if there are any other error messages then your keystore is not setup properly.
5.
@apachekafka
and consumer.
If client authentication is not required in the broker, then the following is a minimal configuration example:
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234
If client authentication is required, then a keystore must be created like in step 1 and the following must also be configured:
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
Other configuration settings that may also be needed depending on our requirements and the broker configuration:
1. ssl.provider (Optional). The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
2. ssl.cipher.suites (Optional). A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to
negotiate the security settings for a network connection using TLS or SSL network protocol.
3. ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1. It should list at least one of the protocols configured on the broker side
4. ssl.truststore.type=JKS
5. ssl.keystore.type=JKS
Examples using console-producer and console-consumer:
-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 84 of 105
Implementation
4. Configure a SASL port in server.properties, by adding at least one of SASL_PLAINTEXT or SASL_SSL to the listeners parameter, which contains
Operations
Security
listeners=SASL_PLAINTEXT://host.name:port
Kafka Connect
Kafka Streams
If SASL_SSL is used, then SSL must also be configured. If you are only configuring a SASL port (or if you want the Kafka brokers to authenticate
PERFORMANCE
each other using SASL) then make sure you set the same SASL protocol for inter-broker communication:
POWERED BY
security.inter.broker.protocol=SASL_PLAINTEXT (or SASL_SSL)
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
sasl.enabled.mechanisms=GSSAPI (,PLAIN)
CONTACT US
APACHE
6. Configure the SASL mechanism for inter-broker communication in server.properties if using SASL for inter-broker communication:
Download
@apachekafka
7. Follow the steps in GSSAPI (Kerberos) or PLAIN to configure SASL for the enabled mechanisms. To enable multiple mechanisms in the broker,
follow the steps here.
Important notes:
1. KafkaServer is the section name in the JAAS file used by each KafkaServer/Broker. This section provides SASL configuration options
for the broker including any SASL client connections made by the broker for inter-broker communication.
2. Client section is used to authenticate a SASL connection with zookeeper. It also allows the brokers to set SASL ACL on zookeeper
nodes which locks these nodes down so that only the brokers can modify it. It is necessary to have the same principal name across all
brokers. If you want to use a section name other than Client, set the system property zookeeper.sasl.client to the appropriate name
(e.g., -Dzookeeper.sasl.client=ZkClient).
3. ZooKeeper uses "zookeeper" as the service name by default. If you want to change this, set the system property
zookeeper.sasl.client.username to the appropriate name (e.g., -Dzookeeper.sasl.client.username=zk).
2.
-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
5. Follow the steps in GSSAPI (Kerberos) or PLAIN to configure SASL for the selected mechanism.
3.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 85 of 105
and configure it (Ubuntu, Redhat). Note that if you are using Oracle Java, you will need to download JCE policy files for your Java version
and copy them to $JAVA_HOME/jre/lib/security.
2. Create Kerberos Principals
If you are using the organization's Kerberos or Active Directory server, ask your Kerberos administrator for a principal for each Kafka
Security
broker in your cluster and for every operating system user that will access Kafka with Kerberos authentication (via clients and tools).
Kafka Connect
If you have installed your own Kerberos, you will need to create these principals yourself using the following commands:
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
3. Make sure all hosts can be reachable using hostnames - it is a Kerberos requirement that all your hosts can be resolved with their FQDNs.
2. Configuring Kafka Brokers
1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for
this example (note that each broker should have its own keytab):
CONTACT US
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
APACHE
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab"
Download
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};
@apachekafka
KafkaServer section in the JAAS file tells the broker which principal to use and the location of the keytab where this principal is stored.
It allows the broker to login using the keytab specified in this section. See notes for more details on Zookeeper SASL configuration.
2. Pass the JAAS and optionally the krb5 file locations as JVM parameters to each Kafka broker (see here for more details):
-Djava.security.krb5.conf=/etc/kafka/krb5.conf
-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
3. Make sure the keytabs configured in the JAAS file are readable by the operating system user who is starting kafka broker.
4. Configure SASL port and SASL mechanisms in server.properties as described here. For example:
listeners=SASL_PLAINTEXT://host.name:port
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
We must also configure the service name in server.properties, which should match the principal name of the kafka brokers. In the above
example, principal is "kafka/kafka1.hostname.com@EXAMPLE.com", so:
sasl.kerberos.service.name=kafka
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 86 of 105
1. Clients (producers, consumers, connect workers, etc) will authenticate to the cluster with their own principal (usually with the same name
Implementation
as the user running the client), so obtain or create these principals as needed. Then create a JAAS file for each principal. The KafkaClient
section describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration
Operations
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
Kafka Streams
useKeyTab=true
PERFORMANCE
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.keytab"
POWERED BY
principal="kafka-client-1@EXAMPLE.COM";
};
PROJECT INFO
ECOSYSTEM
For command-line utilities like kafka-console-consumer or kafka-console-producer, kinit can be used along with "useTicketCache=true" as
CLIENTS
in:
EVENTS
KafkaClient {
CONTACT US
com.sun.security.auth.module.Krb5LoginModule required
APACHE
useTicketCache=true;
};
Download
2. Pass the JAAS and optionally krb5 file locations as JVM parameters to each client JVM (see here for more details):
@apachekafka
-Djava.security.krb5.conf=/etc/kafka/krb5.conf
-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
3. Make sure the keytabs configured in the kafka_client_jaas.conf are readable by the operating system user who is starting kafka client.
4. Configure the following properties in producer.properties or consumer.properties:
4.
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret";
};
This configuration defines two users (admin and alice). The properties username and password in the KafkaServer section are used
by the broker to initiate connections to other brokers. In this example, admin is the user for inter-broker communication. The set of
properties user_userName defines the passwords for all users that connect to the broker and the broker validates all client connections
including those from other brokers using these properties.
2. Pass the JAAS config file location as JVM parameter to each Kafka broker:
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 87 of 105
-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
Implementation
Operations
Security
3. Configure SASL port and SASL mechanisms in server.properties as described here. For example:
Kafka Connect
listeners=SASL_SSL://host.name:port
Kafka Streams
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
PERFORMANCE
sasl.enabled.mechanisms=PLAIN
POWERED BY
PROJECT INFO
2. Configuring Kafka Clients
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
APACHE
username="alice"
password="alice-secret";
};
Download
@apachekafka
The properties username and password in the KafkaClient section are used by clients to configure the user for client connections. In
this example, clients connect to the broker as user alice.
2. Pass the JAAS config file location as JVM parameter to each client JVM:
-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
security.provider.n=providerClassName
where providerClassName is the fully qualified name of the new provider and n is the preference order with lower numbers indicating
higher preference.
Alternatively, you can register providers dynamically at runtime by invoking Security.addProvider at the beginning of the client
application or in a static initializer in the login module. For example:
Security.addProvider(new PlainSaslServerProvider());
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Page 88 of 105
Implementation
Operations
Security
5.
Kafka Connect
1. Specify configuration for the login modules of all enabled mechanisms in the KafkaServer section of the JAAS config file. For example:
Kafka Streams
PERFORMANCE
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
POWERED BY
useKeyTab=true
storeKey=true
PROJECT INFO
keyTab="/etc/security/keytabs/kafka_server.keytab"
ECOSYSTEM
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
CLIENTS
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
EVENTS
password="admin-secret"
CONTACT US
user_admin="admin-secret"
user_alice="alice-secret";
APACHE
};
Download
@apachekafka
sasl.enabled.mechanisms=GSSAPI,PLAIN
3. Specify the SASL security protocol and mechanism for inter-broker communication in server.properties if required:
4. Follow the mechanism-specific steps in GSSAPI (Kerberos) and PLAIN to configure SASL for the enabled mechanisms.
6.
allow.everyone.if.no.acl.found=true
One can also add super users in broker.properties like the following (note that the delimiter is semicolon since SSL user names may contain comma).
super.users=User:Bob;User:Alice
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 89 of 105
By default, the SSL user name will be of the form "CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown". One can change that by
setting a customized PrincipalBuilder in broker.properties like the following.
Operations
principal.builder.class=CustomizedPrincipalBuilderClass
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
By default, the SASL user name will be the primary part of the Kerberos principal. One can change that by setting
sasl.kerberos.principal.to.local.rules to a customized rule in broker.properties. The format of
sasl.kerberos.principal.to.local.rules is a list where each rule works in the same way as the auth_to_local in Kerberos configuration file
(krb5.conf). Each rules starts with RULE: and contains an expression in the format [n:string](regexp)s/pattern/replacement/g. See the kerberos
documentation for more details. An example of adding a rule to properly translate user@MYDOMAIN.COM to user while also keeping the default rule in place
PROJECT INFO
is:
ECOSYSTEM
sasl.kerberos.principal.to.local.rules=RULE:[1:$1@$0](.*@MYDOMAIN.COM)s/@.*//,DEFAULT
CLIENTS
EVENTS
CONTACT US
APACHE
Download
@apachekafka
OPTION
DESCRIPTION
--add
Action
--remove
Action
--list
--authorizer
--authorizer-prope
rties
--cluster
--topic [topic-nam
e]
--group [group-na
me]
DEFAULT
OPTION TYPE
Action
kafka.security.auth.SimpleAclAuthoriz
er
Configuration
Configuration
host:2181
Specifies cluster as resource.
Resource
Resource
Resource
th Allow permission.
Principal
th Deny permission.
Principal
--deny-host
ccess.
ed access.
Host
Host
Valid values are : Read, Write, Create, Delete, Alter, Describe, ClusterA
All
Operation
ction, All
Convenience option to add/remove acls for producer role. This will ge
--producer
nerate acls that allows WRITE, DESCRIBE on topic and CREATE on clu
Convenience
ster.
Convenience option to add/remove acls for consumer role. This will g
--consumer
enerate acls that allows READ, DESCRIBE on topic and READ on cons
Convenience
umer-group.
--force
https://kafka.apache.org/documentation
Convenience
11/4/2016
Apache Kafka
Implementation
Operations
Page 90 of 105
Examples
Adding Acls
Suppose you want to add an acl "Principals User:Bob and User:Alice are allowed to perform Operation Read and Write on Topic Test-Topic from IP
Security
198.51.100.0 and IP 198.51.100.1". You can do that by executing the CLI with following options:
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
By default, all principals that don't have an explicit acl that allows access for an operation to a resource are denied. In rare cases where an allow acl is
defined that allows access to all but some principal we will have to use the --deny-principal and --deny-host option. For example, if we want to allow all
users to Read from Test-topic but only deny User:BadBob from IP 198.51.100.3 we can do so using following commands:
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Note that ``--allow-host`` and ``deny-host`` only support IP addresses (hostnames are not supported). Above examples add acls to a topic by specifying
--topic [topic-name] as the resource option. Similarly user can add acls to cluster by specifying --cluster and to a consumer group by specifying --group
[group-name].
Removing Acls
Removing acls is pretty much the same. The only difference is instead of --add option users will have to specify --remove option. To remove the acls added
Download
@apachekafka
by the first example above we can execute the CLI with following options:
List Acls
We can list acls for any resource by specifying the --list option with the resource. To list all acls for Test-topic we can execute the CLI with following
options:
Similarly to add Alice as a consumer of Test-topic with consumer group Group-1 we just have to pass --consumer option:
Note that for consumer option we must also specify the consumer group. In order to remove a principal from producer or consumer role we just need to
pass --remove option.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 91 of 105
As an example, say we wish to encrypt both broker-client and broker-broker communication with SSL. In the first incremental bounce, a SSL port is opened on
each node:
Operations
listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092
Security
Kafka Connect
Kafka Streams
We then restart the clients, changing their config to point at the newly opened, secured port:
PERFORMANCE
bootstrap.servers = [broker1:9092,...]
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
security.protocol = SSL
...etc
In the second incremental server bounce we instruct Kafka to use SSL as the broker-broker protocol (which will use the same SSL port):
EVENTS
CONTACT US
APACHE
listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092
security.inter.broker.protocol=SSL
In the final bounce we secure the cluster by closing the PLAINTEXT port:
Download
listeners=SSL://broker1:9092
security.inter.broker.protocol=SSL
@apachekafka
Alternatively we might choose to open multiple ports so that different protocols can be used for broker-broker and broker-client communication. Say we
wished to use SSL encryption throughout (i.e. for broker-broker and broker-client communication) but we'd like to add SASL authentication to the broker-client
connection also. We would achieve this by opening two additional ports during the first bounce:
listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092,SASL_SSL://broker1:9093
We would then restart the clients, changing their config to point at the newly opened, SASL & SSL secured port:
bootstrap.servers = [broker1:9093,...]
security.protocol = SASL_SSL
...etc
The second server bounce would switch the cluster to use encrypted broker-broker communication via the SSL port we previously opened on port 9092:
listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092,SASL_SSL://broker1:9093
security.inter.broker.protocol=SSL
The final bounce secures the cluster by closing the PLAINTEXT port.
listeners=SSL://broker1:9092,SASL_SSL://broker1:9093
security.inter.broker.protocol=SSL
ZooKeeper can be secured independently of the Kafka cluster. The steps for doing this are covered in section 7.6.2.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 92 of 105
The metadata stored in ZooKeeper for the Kafka cluster is world-readable, but can only be modified by the brokers. The rationale behind this decision is that
the data stored in ZooKeeper is not sensitive, but inappropriate manipulation of that data can cause cluster disruption. We also recommend limiting the
access to ZooKeeper via network segmentation (only brokers and some admin tools need access to ZooKeeper if the new Java consumer and producer
clients are used).
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
set to secure. This tool traverses the corresponding sub-trees changing the ACLs of the znodes
It is also possible to turn off authentication in a secure cluster. To do it, follow these steps:
1. Perform a rolling restart of brokers setting the JAAS login file, which enables brokers to authenticate, but setting zookeeper.set.acl to false. At the
end of the rolling restart, brokers stop creating znodes with secure ACLs, but are still able to authenticate and manipulate all znodes
2. Execute the ZkSecurityMigrator tool. To execute the tool, run this script ./bin/zookeeper-security-migration.sh with zookeeper.acl
unsecure. This tool traverses the corresponding sub-trees changing the ACLs of the znodes
Download
3. Perform a second rolling restart of brokers, this time omitting the system property that sets the JAAS login file
Here is an example of how to run the migration tool:
@apachekafka
./bin/zookeeper-security-migration --help
8. KAFKA CONNECT
8.1 Overview
Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors
that move large collections of data into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all your application servers into
Kafka topics, making the data available for stream processing with low latency. An export job can deliver data from Kafka topics into secondary storage and
query systems or into batch systems for offline analysis. Kafka Connect features include:
A common framework for Kafka connectors - Kafka Connect standardizes integration of other data systems with Kafka, simplifying connector
development, deployment, and management
Distributed and standalone modes - scale up to a large, centrally managed service supporting an entire organization or scale down to development,
testing, and small production deployments
REST interface - submit and manage connectors to your Kafka Connect cluster via an easy to use REST API
Automatic offset management - with just a little information from connectors, Kafka Connect can manage the offset commit process automatically so
connector developers do not need to worry about this error prone part of connector development
Distributed and scalable by default - Kafka Connect builds on the existing group management protocol. More workers can be added to scale up a Kafka
Connect cluster.
Streaming/batch integration - leveraging Kafka's existing capabilities, Kafka Connect is an ideal solution for bridging streaming and batch data systems
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 93 of 105
Running Kafka Connect
Kafka Connect currently supports two modes of execution: standalone (single process) and distributed. In standalone mode all work is performed in a single
process. This configuration is simpler to setup and get started with and may be useful in situations where only one worker makes sense (e.g. collecting log
Security
files), but it does not benefit from some of the features of Kafka Connect such as fault tolerance. You can start a standalone process with the following
Kafka Connect
command:
Kafka Streams
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
The first parameter is the configuration for the worker. This includes settings such as the Kafka connection parameters, serialization format, and how
frequently to commit offsets. The provided example should work well with a local cluster running with the default configuration provided by
config/server.properties . It will require tweaking to use with a different configuration or production deployment. All workers (both standalone and
distributed) require a few configs:
key.converter - Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the
format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any
CONTACT US
APACHE
value.converter - Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the
format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any
serialization format. Examples of common formats include JSON and Avro.
Download
@apachekafka
The remaining parameters are connector configuration files. You may include as many as you want, but all will execute within the same process (on different
threads). Distributed mode handles automatic balancing of work, allows you to scale up (or down) dynamically, and offers fault tolerance both in the active
tasks and for configuration and offset commit data. Execution is very similar to standalone mode:
The difference is in the class which is started and the configuration parameters which change how the Kafka Connect process decides where to store
configurations, how to assign work, and where to store offsets and task statues. In the distributed mode, Kafka Connect stores the offsets, configs and task
statuses in Kafka topics. It is recommended to manually create the topics for offset, configs and statuses in order to achieve the desired the number of
partitions and replication factors. If the topics are not yet created when starting Kafka Connect, the topics will be auto created with default number of
partitions and replication factor, which may not be best suited for its usage. In particular, the following configuration parameters, in addition to the common
settings mentioned above, are critical to set before starting your cluster:
group.id (default connect-cluster ) - unique name for the cluster, used in forming the Connect cluster group; note that this must not conflict
with consumer group IDs
config.storage.topic (default connect-configs ) - topic to use for storing connector and task configurations; note that this should be a
single partition, highly replicated, compacted topic. You may need to manually create the topic to ensure the correct configuration as auto created topics
may have multiple partitions or be automatically configured for deletion rather than compaction
offset.storage.topic (default connect-offsets ) - topic to use for storing offsets; this topic should have many partitions, be replicated, and
be configured for compaction
status.storage.topic (default connect-status ) - topic to use for storing statuses; this topic can have multiple partitions, and should be
replicated and configured for compaction
Note that in distributed mode the connector configurations are not passed on the command line. Instead, use the REST API described below to create, modify,
and destroy connectors.
Configuring Connectors
Connector configurations are simple key-value mappings. For standalone mode these are defined in a properties file and passed to the Connect process on
the command line. In distributed mode, they will be included in the JSON payload for the request that creates (or modifies) the connector. Most
configurations are connector dependent, so they can't be outlined here. However, there are a few common options:
name - Unique name for the connector. Attempting to register again with the same name will fail.
tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this
level of parallelism.
key.converter - (optional) Override the default key converter set by the worker.
value.converter - (optional) Override the default value converter set by the worker.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 94 of 105
The connector.class config supports several formats: the full name or alias of the class for this connector. If the connector is
org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name or use FileStreamSink or FileStreamSinkConnector to make the
configuration a bit shorter. Sink connectors also have one additional option to control their input:
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
For any other options, you should consult the documentation for the connector.
REST API
Since Kafka Connect is intended to be run as a service, it also provides a REST API for managing connectors. By default, this service runs on port 8083. The
following are the currently supported endpoints:
@apachekafka
GET /connectors/{name}/status - get current status of the connector, including if it is running, failed, paused, etc., which worker it is assigned to,
error information if it has failed, and the state of all its tasks
Download
PUT /connectors/{name}/pause - pause the connector and its tasks, which stops message processing until the connector is resumed
PUT /connectors/{name}/resume - resume a paused connector (or do nothing if the connector is not paused)
DELETE /connectors/{name} - delete a connector, halting all tasks and deleting its configuration
Kafka Connect also provides a REST API for getting information about connector plugins:
GET /connector-plugins - return a list of connector plugins installed in the Kafka Connect cluster. Note that the API only checks for connectors on
the worker that handles the request, which means you may see inconsistent results, especially during a rolling upgrade if you add new connector jars
PUT /connector-plugins/{connector-type}/config/validate - validate the provided configuration values against the configuration
definition. This API performs per config validation, returns suggested values and error messages during validation.
To copy data between Kafka and another system, users create a Connector for the system they want to pull data from or push data to. Connectors come
in two flavors: SourceConnectors import data from another system (e.g. JDBCSourceConnector would import a relational database into Kafka)
and SinkConnectors export data (e.g. HDFSSinkConnector would export the contents of a Kafka topic to an HDFS file). Connectors
perform any data copying themselves: their configuration describes the data to be copied, and the Connector is responsible for breaking that job into a
set of Tasks that can be distributed to workers. These Tasks also come in two corresponding flavors: SourceTask and SinkTask
assignment in hand, each Task must copy its subset of the data to or from Kafka. In Kafka Connect, it should always be possible to frame these
assignments as a set of input and output streams consisting of records with consistent schemas. Sometimes this mapping is obvious: each file in a set of log
files can be considered a stream with each parsed line forming a record using the same schema and offsets stored as byte offsets in the file. In other cases it
may require more effort to map to this model: a JDBC connector can map each table to a stream, but the offset is less clear. One possible mapping uses a
timestamp column to generate queries incrementally returning new data, and the last queried timestamp can be used as the offset.
Streams and Records
Each stream should be a sequence of key-value records. Both the keys and values can have complex structure -- many primitive types are provided, but arrays,
objects, and nested data structures can be represented as well. The runtime data format does not assume any particular serialization format; this conversion
is handled internally by the framework. In addition to the key and value, records (both those generated by sources and those delivered to sinks) have
associated stream IDs and offsets. These are used by the framework to periodically commit the offsets of data that have been processed so that in the event
of failures, processing can resume from the last committed offsets, avoiding unnecessary reprocessing and duplication of events.
Dynamic Connectors
Not all jobs are static, so Connector implementations are also responsible for monitoring the external system for any changes that might require
reconfiguration. For example, in the JDBCSourceConnector example, the Connector might assign a set of tables to each Task . When a new table
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 95 of 105
is created, it must discover this so it can assign the new table to one of the Tasks by updating its configuration. When it notices a change that requires
reconfiguration (or a change in the number of Tasks ), it notifies the framework and the framework updates any corresponding Tasks .
Operations
Security
Kafka Connect
Developing a connector only requires implementing two interfaces, the Connector and Task . A simple example is included with the source code for
Kafka Streams
Kafka in the file package. This connector is meant for use in standalone mode and has implementations of a SourceConnector / SourceTask
PERFORMANCE
POWERED BY
read each line of a file and emit it as a record and a SinkConnector / SinkTask that writes each record to a file. The rest of this section will walk
through some code to demonstrate the key steps in creating a connector, but developers should also refer to the full example source code as many details
are omitted for brevity.
PROJECT INFO
Connector Example
ECOSYSTEM
We'll cover the SourceConnector as a simple example. SinkConnector implementations are very similar. Start by creating the class that inherits
from SourceConnector and add a couple of fields that will store parsed configuration information (the filename to read from and the topic to send data
CLIENTS
to):
EVENTS
CONTACT US
APACHE
Download
@apachekafka
The easiest method to fill in is getTaskClass() , which defines the class that should be instantiated in worker processes to actually read the data:
@Override
public Class<? extends Task> getTaskClass() {
return FileStreamSourceTask.class;
}
We will define the FileStreamSourceTask class below. Next, we add some standard lifecycle methods, start() and stop() :
@Override
public void start(Map<String, String> props) {
// The complete version includes error handling as well.
filename = props.get(FILE_CONFIG);
topic = props.get(TOPIC_CONFIG);
}
@Override
public void stop() {
// Nothing to do since no background monitoring is required.
}
Finally, the real core of the implementation is in taskConfigs() . In this case we are only handling a single file, so even though we may be permitted to
generate more tasks as per the maxTasks argument, we return a list with only one entry:
@Override
public List<Map<String, String>> taskConfigs(int maxTasks) {
ArrayList<Map<String, String>> configs = new ArrayList<>();
// Only one input stream makes sense.
Map<String, String> config = new HashMap<>();
if (filename != null)
config.put(FILE_CONFIG, filename);
config.put(TOPIC_CONFIG, topic);
configs.add(config);
return configs;
}
Although not used in the example, SourceTask also provides two APIs to commit offsets in the source system: commit and commitRecord
APIs are provided for source systems which have an acknowledgement mechanism for messages. Overriding these methods allows the source connector to
acknowledge messages in the source system, either in bulk or individually, once they have been written to Kafka. The commit API stores the offsets in the
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 96 of 105
source system, up to the offsets that have been returned by poll . The implementation of this API should block until the commit is complete. The
commitRecord API saves the offset in the source system for each SourceRecord after it is written to Kafka. As Kafka Connect will record offsets
automatically, SourceTask s are not required to implement them. In cases where a connector does need to acknowledge messages in the source system,
only one of the APIs is typically required. Even with multiple tasks, this method implementation is usually pretty simple. It just has to determine the number of
Security
input tasks, which may require contacting the remote service it is pulling data from, and then divvy them up. Because some patterns for splitting work among
Kafka Connect
Kafka Streams
PERFORMANCE
tasks are so common, some utilities are provided in ConnectorUtils to simplify these cases. Note that this simple example does not include dynamic
input. See the discussion in the next section for how to trigger updates to task configs.
Task Example - Source Task
POWERED BY
Next we'll describe the implementation of the corresponding SourceTask . The implementation is short, but too long to cover completely in this guide.
PROJECT INFO
to create a class inheriting from the appropriate base Task class. It also has some standard lifecycle methods:
We'll use pseudo-code to describe most of the implementation, but you can refer to the source code for the full example. Just as with the connector, we need
ECOSYSTEM
public class FileStreamSourceTask extends SourceTask {
CLIENTS
String filename;
InputStream stream;
EVENTS
String topic;
CONTACT US
@Override
APACHE
Download
}
@apachekafka
@Override
public synchronized void stop() {
stream.close();
}
These are slightly simplified versions, but show that that these methods should be relatively simple and the only work they should perform is allocating or
freeing resources. There are two points to note about this implementation. First, the start() method does not yet handle resuming from a previous
offset, which will be addressed in a later section. Second, the stop() method is synchronized. This will be necessary because SourceTasks
a dedicated thread which they can block indefinitely, so they need to be stopped with a call from a different thread in the Worker. Next, we implement the main
functionality of the task, the poll() method which gets events from the input system and returns a List<SourceRecord> :
@Override
public List<SourceRecord> poll() throws InterruptedException {
try {
ArrayList<SourceRecord> records = new ArrayList<>();
while (streamValid(stream) && records.isEmpty()) {
LineAndOffset line = readToNextLine(stream);
if (line != null) {
Map<String, Object> sourcePartition = Collections.singletonMap("filename", filename);
Map<String, Object> sourceOffset = Collections.singletonMap("position", streamOffset);
records.add(new SourceRecord(sourcePartition, sourceOffset, topic, Schema.STRING_SCHEMA, line));
} else {
Thread.sleep(1);
}
}
return records;
} catch (IOException e) {
// Underlying stream was killed, probably as a result of calling stop. Allow to return
// null, and driving thread will handle any shutdown if necessary.
}
return null;
}
Again, we've omitted some details, but we can see the important steps: the poll() method is going to be called repeatedly, and for each call it will loop
trying to read records from the file. For each line it reads, it also tracks the file offset. It uses this information to create an output SourceRecord
pieces of information: the source partition (there is only one, the single file being read), source offset (byte offset in the file), output topic name, and output
value (the line, and we include a schema indicating this value will always be a string). Other variants of the SourceRecord constructor can also include a
specific output partition and a key. Note that this implementation uses the normal Java InputStream interface and may sleep if data is not available. This
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Page 97 of 105
is acceptable because Kafka Connect provides each task with a dedicated thread. While task implementations have to conform to the basic poll()
interface, they have a lot of flexibility in how they are implemented. In this case, an NIO-based implementation would be more efficient, but this simple
Operations
approach works, is quick to implement, and is compatible with older versions of Java.
Security
Sink Tasks
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
The previous section described how to implement a simple SourceTask . Unlike SourceConnector and SinkConnector , SourceTask
SinkTask have very different interfaces because SourceTask uses a pull interface and SinkTask uses a push interface. Both share the common
lifecycle methods, but the SinkTask interface is quite different:
PROJECT INFO
this.context = context;
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
The SinkTask documentation contains full details, but this interface is nearly as simple as the SourceTask . The put() method should contain
most of the implementation, accepting sets of SinkRecords , performing any required translation, and storing them in the destination system. This
Download
method does not need to ensure the data has been fully written to the destination system before returning. In fact, in many cases internal buffering will be
useful so an entire batch of records can be sent at once, reducing the overhead of inserting events into the downstream data store. The SinkRecords
contain essentially the same information as SourceRecords : Kafka topic, partition, offset and the event key and value. The flush() method is used
@apachekafka
during the offset commit process, which allows tasks to recover from failures and resume from a safe point such that no events will be missed. The method
should push any outstanding data to the destination system and then block until the write has been acknowledged. The offsets parameter can often be
ignored, but is useful in some cases where implementations want to store offset information in the destination store to provide exactly-once delivery. For
example, an HDFS connector could do this and use atomic move operations to make sure the flush() operation atomically commits the data and offsets
to a final location in HDFS.
Resuming from Previous Offsets
The SourceTask implementation included a stream ID (the input filename) and offset (position in the file) with each record. The framework uses this to
commit offsets periodically so that in the case of a failure, the task can recover and minimize the number of events that are reprocessed and possibly
duplicated (or to resume from the most recent offset if Kafka Connect was stopped gracefully, e.g. in standalone mode or due to a job reconfiguration). This
commit process is completely automated by the framework, but only the connector knows how to seek back to the right position in the input stream to
resume from that location. To correctly resume upon startup, the task can use the SourceContext passed into its initialize() method to access
the offset data. In initialize() , we would add a bit more code to read the offset (if it exists) and seek to that position:
Of course, you might need to read many keys for each of the input streams. The OffsetStorageReader interface also allows you to issue bulk reads to
efficiently load all offsets, then apply them by seeking each input stream to the appropriate position.
if (inputsChanged())
this.context.requestTaskReconfiguration();
The framework will promptly request new configuration information and update the tasks, allowing them to gracefully commit their progress before
reconfiguring them. Note that in the SourceConnector this monitoring is currently left up to the connector implementation. If an extra thread is required
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 98 of 105
to perform this monitoring, the connector must allocate it itself. Ideally this code for monitoring changes would be isolated to the Connector
would not need to worry about them. However, changes can also affect tasks, most commonly when one of their input streams is destroyed in the input
system, e.g. if a table is dropped from a database. If the Task encounters the issue before the Connector , which will be common if the
needs to poll for changes, the Task will need to handle the subsequent error. Thankfully, this can usually be handled simply by catching and handling the
Security
appropriate exception. SinkConnectors usually only have to handle the addition of streams, which may translate to new entries in their outputs (e.g., a
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
new database table). The framework manages any changes to the Kafka input, such as when the set of input topics changes because of a regex subscription.
SinkTasks should expect new input streams, which may require creating new resources in the downstream system, such as a new table in a database.
The trickiest situation to handle in these cases may be conflicts between multiple SinkTasks seeing a new input stream for the first time and
simultaneously trying to create the new resource. SinkConnectors , on the other hand, will generally require no special code for handling a dynamic set of
streams.
EVENTS
private static final ConfigDef CONFIG_DEF = new ConfigDef()
CONTACT US
Download
return CONFIG_DEF;
}
@apachekafka
ConfigDef class is used for specifying the set of expected configurations. For each configuration, you can specify the name, the type, the default value,
the documentation, the group information, the order in the group, the width of the configuration value and the name suitable for display in the UI. Plus, you can
provide special validation logic used for single configuration validation by overriding the Validator class. Moreover, as there may be dependencies
between configurations, for example, the valid values and visibility of a configuration may change according to the values of other configurations. To handle
this, ConfigDef allows you to specify the dependents of a configuration and to provide an implementation of Recommender to get valid values and set
visibility of a configuration given the current configuration values. Also, the validate() method in Connector provides a default validation
implementation which returns a list of allowed configurations together with configuration errors and recommended values for each configuration. However, it
does not use the recommended values for configuration validation. You may provide an override of the default implementation for customized configuration
validation, which may use the recommended values.
If you are implementing a source connector, you'll need to decide when and how to create schemas. Where possible, you should avoid recomputing them as
much as possible. For example, if your connector is guaranteed to have a fixed schema, create it statically and reuse a single instance. However, many
connectors will have dynamic schemas. One simple example of this is a database connector. Considering even just a single table, the schema will not be
predefined for the entire connector (as it varies from table to table). But it also may not be fixed for a single table over the lifetime of the connector since the
user may execute an ALTER TABLE command. The connector must be able to detect these changes and react appropriately. Sink connectors are usually
simpler because they are consuming data and therefore do not need to create schemas. However, they should take just as much care to validate that the
schemas they receive have the expected format. When the schema does not match -- usually indicating the upstream producer is generating invalid data that
cannot be correctly translated to the destination system -- sink connectors should throw an exception to indicate this error to the system.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Page 99 of 105
Kafka Connect Administration
Kafka Connect's REST layer provides a set of APIs to enable administration of the cluster. This includes APIs to view the configuration of connectors and the
status of their tasks, as well as to alter their current behavior (e.g. changing configuration and restarting tasks).
Security
Kafka Connect
Kafka Streams
PERFORMANCE
When a connector is first submitted to the cluster, the workers rebalance the full set of connectors in the cluster and their tasks so that each worker has
approximately the same amount of work. This same rebalancing procedure is also used when connectors increase or decrease the number of tasks they
require, or when a connector's configuration is changed. You can use the REST API to view the current status of a connector and its tasks, including the id of
the worker to which each was assigned. For example, querying the status of a file source (using GET /connectors/file-source/status
produce output like the following:
POWERED BY
PROJECT INFO
{
"name": "file-source",
"connector": {
ECOSYSTEM
"state": "RUNNING",
CLIENTS
"worker_id": "192.168.1.208:8083"
},
EVENTS
"tasks": [
{
CONTACT US
"id": 0,
APACHE
"state": "RUNNING",
"worker_id": "192.168.1.209:8083"
}
Download
]
}
@apachekafka
Connectors and their tasks publish status updates to a shared topic (configured with status.storage.topic ) which all workers in the cluster monitor.
Because the workers consume this topic asynchronously, there is typically a (short) delay before a state change is visible through the status API. The
following states are possible for a connector or one of its tasks:
UNASSIGNED: The connector/task has not yet been assigned to a worker.
RUNNING: The connector/task is running.
PAUSED: The connector/task has been administratively paused.
FAILED: The connector/task has failed (usually by raising an exception, which is reported in the status output).
In most cases, connector and task states will match, though they may be different for short periods of time when changes are occurring or if tasks have
failed. For example, when a connector is first started, there may be a noticeable delay before the connector and its tasks have all transitioned to the RUNNING
state. States will also diverge when tasks fail since Connect does not automatically restart failed tasks. To restart a connector/task manually, you can use the
restart APIs listed above. Note that if you try to restart a task while a rebalance is taking place, Connect will return a 409 (Conflict) status code. You can retry
after the rebalance completes, but it might not be necessary since rebalances effectively restart all the connectors and tasks in the cluster.
It's sometimes useful to temporarily stop the message processing of a connector. For example, if the remote system is undergoing maintenance, it would be
preferable for source connectors to stop polling it for new data instead of filling logs with exception spam. For this use case, Connect offers a pause/resume
API. While a source connector is paused, Connect will stop polling it for additional records. While a sink connector is paused, Connect will stop pushing new
messages to it. The pause state is persistent, so even if you restart the cluster, the connector will not begin message processing again until the task has been
resumed. Note that there may be a delay before all of a connector's tasks have transitioned to the PAUSED state since it may take time for them to finish
whatever processing they were in the middle of when being paused. Additionally, failed tasks will not transition to the PAUSED state until they have been
restarted.
9. KAFKA STREAMS
9.1 Overview
Kafka Streams is a client library for processing and analyzing data stored in Kafka and either write the resulting data back to Kafka or send the final output to
an external system. It builds upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing
support, and simple yet efficient management of application state. Kafka Streams has a low barrier to entry: You can quickly write and run a small-scale
proof-of-concept on a single machine; and you only need to run additional instances of your application on multiple machines to scale up to high-volume
production workloads. Kafka Streams transparently handles the load balancing of multiple instances of the same application by leveraging Kafka's
parallelism model.
Some highlights of Kafka Streams:
Designed as a simple and lightweight client library, which can be easily embedded in any Java application and integrated with any existing packaging,
deployment and operational tools that users have for their streaming applications.
Has no external dependencies on systems other than Apache Kafka itself as the internal messaging layer; notably, it uses Kafka's partitioning model to
horizontally scale processing while maintaining strong ordering guarantees.
Supports fault-tolerant local state, which enables very fast and efficient stateful operations like joins and windowed aggregations.
Employs one-record-at-a-time processing to achieve low processing latency, and supports event-time based windowing operations.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Kafka Streams
Core Concepts
PERFORMANCE
We first summarize the key concepts of Kafka Streams.
POWERED BY
PROJECT INFO
ECOSYSTEM
A stream is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an
ordered, replayable, and fault-tolerant sequence of immutable data records, where a data record is defined as a key-value pair.
CLIENTS
A stream processing application written in Kafka Streams defines its computational logic through one or more processor topologies, where a processor
EVENTS
A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a
topology is a graph of stream processors (nodes) that are connected by streams (edges).
CONTACT US
APACHE
time from its upstream processors in the topology, applying its operation to it, and may subsequently producing one or more output records to its
downstream processors.
Kafka Streams offers two ways to define the stream processing topology: the Kafka Streams DSL provides the most common data transformation operations
such as map and filter ; the lower-level Processor API allows developers define and connect custom processors as well as to interact with
Download
stores.
Time
@apachekafka
A critical aspect in stream processing is the notion of time, and how it is modeled and integrated. For example, some operations such as windowing
defined based on time boundaries.
Common notions of time in streams are:
Event time - The point in time when an event or data record occurred, i.e. was originally created "at the source".
Processing time - The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is
being consumed. The processing time may be milliseconds, hours, or days etc. later than the original event time.
Ingestion time - The point in time when an event or data record is stored in a topic partition by a Kafka broker. The difference to event time is that this
ingestion timestamp is generated when the record is appended to the target topic by the Kafka broker, not when the record is created "at the source". The
difference to processing time is that processing time is when the stream processing application processes the record. For example, if a record is never
processed, there is no notion of processing time for it, but it still has an ingestion time.
The choice between event-time and ingestion-time is actually done through the configuration of Kafka (not Kafka Streams): From Kafka 0.10.x onwards,
timestamps are automatically embedded into Kafka messages. Depending on Kafka's configuration these timestamps represent event-time or ingestion-time.
The respective Kafka configuration setting can be specified on the broker level or per topic. The default timestamp extractor in Kafka Streams will retrieve
these embedded timestamps as-is. Hence, the effective time semantics of your application depend on the effective Kafka configuration for these embedded
timestamps.
Kafka Streams assigns a timestamp to every data record via the TimestampExtractor interface. Concrete implementations of this interface may
retrieve or compute timestamps based on the actual contents of data records such as an embedded timestamp field to provide event-time semantics, or use
any other approach such as returning the current wall-clock time at the time of processing, thereby yielding processing-time semantics to stream processing
applications. Developers can thus enforce different notions of time depending on their business needs. For example, per-record timestamps describe the
progress of a stream with regards to time (although records may be out-of-order within the stream) and are leveraged by time-dependent operations such as
joins.
Finally, whenever a Kafka Streams application writes records to Kafka, then it will also assign timestamps to these new records. The way the timestamps are
assigned depends on the context:
When new output records are generated via processing some input record, for example, context.forward() triggered in the process()
call, output record timestamps are inherited from input record timestamps directly.
When new output records are generated via periodic functions such as punctuate() , the output record timestamp is defined as the current internal
time (obtained through context.timestamp() ) of the stream task.
For aggregations, the timestamp of a resulting aggregate update record will be that of the latest arrived input record that triggered the update.
States
Some stream processing applications don't require state, which means the processing of a message is independent from the processing of all other
messages. However, being able to maintain state opens up many possibilities for sophisticated stream processing applications: you can join input streams, or
group and aggregate data records. Many such stateful operators are provided by the Kafka Streams DSL.
Kafka Streams provides so-called state stores, which can be used by stream processing applications to store and query data. This is an important capability
when implementing stateful operations. Every task in Kafka Streams embeds one or more state stores that can be accessed via APIs to store and query data
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
As we have mentioned above, the computational logic of a Kafka Streams application is defined as a processor topology. Currently Kafka Streams provides
two sets of APIs to define the processor topology, which will be described in the subsequent sections.
POWERED BY
PROJECT INFO
ECOSYSTEM
Processor
CLIENTS
Developers can define their customized processing logic by implementing the Processor interface, which provides process and punctuate
EVENTS
CONTACT US
methods. The process method is performed on each of the received record; and the punctuate method is performed periodically based on elapsed
time. In addition, the processor can maintain the current ProcessorContext instance variable initialized in the init method, and use the context to
schedule the punctuation period ( context().schedule ), to forward the modified / new key-value pair to downstream processors ( context
().forward ), to commit the current processing progress ( context().commit ), etc.
APACHE
public class MyProcessor extends Processor {
private ProcessorContext context;
Download
@apachekafka
@Override
@SuppressWarnings("unchecked")
public void init(ProcessorContext context) {
this.context = context;
this.context.schedule(1000);
this.kvStore = (KeyValueStore) context.getStateStore("Counts");
}
@Override
public void process(String dummy, String line) {
String[] words = line.toLowerCase().split(" ");
for (String word : words) {
Integer oldValue = this.kvStore.get(word);
if (oldValue == null) {
this.kvStore.put(word, 1);
} else {
this.kvStore.put(word, oldValue + 1);
}
}
}
@Override
public void punctuate(long timestamp) {
KeyValueIterator iter = this.kvStore.all();
while (iter.hasNext()) {
KeyValue entry = iter.next();
context.forward(entry.key, entry.value.toString());
}
iter.close();
context.commit();
}
@Override
public void close() {
this.kvStore.close();
}
};
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
In the punctuate method, iterate the local state store and send the aggregated counts to the downstream processor, and commit the current stream
state.
Kafka Streams
Processor Topology
PERFORMANCE
POWERED BY
With the customized processors defined in the Processor API, developers can use the TopologyBuilder to build a processor topology by connecting
these processors together:
PROJECT INFO
TopologyBuilder builder = new TopologyBuilder();
ECOSYSTEM
builder.addSource("SOURCE", "src-topic")
CLIENTS
EVENTS
.addProcessor("PROCESS1", MyProcessor1::new /* the ProcessorSupplier that can generate MyProcessor1 */, "SOU
.addProcessor("PROCESS2", MyProcessor2::new /* the ProcessorSupplier that can generate MyProcessor2 */, "PRO
CONTACT US
.addProcessor("PROCESS3", MyProcessor3::new /* the ProcessorSupplier that can generate MyProcessor3 */, "PRO
APACHE
Download
@apachekafka
There are several steps in the above code to build the topology, and here is a quick walk through:
First of all a source node named "SOURCE" is added to the topology using the addSource method, with one Kafka topic "src-topic" fed to it.
Three processor nodes are then added using the addProcessor method; here the first processor is a child of the "SOURCE" node, but is the parent of
the other two processors.
Finally three sink nodes are added to complete the topology using the addSink method, each piping from a different parent processor node and writing
to a separate topic.
Local State Store
Note that the Processor API is not limited to only accessing the current records as they arrive, but can also maintain local state stores that keep recently
arrived records to use in stateful processing operations such as aggregation or windowed joins. To take advantage of this local states, developers can use the
TopologyBuilder.addStateStore method when building the processor topology to create the local state and associate it with the processor nodes
that needs to access it; or they can connect a created local state store with the existing processor nodes through
TopologyBuilder.connectProcessorAndStateStores .
.addProcessor("PROCESS2", MyProcessor3::new /* the ProcessorSupplier that can generate MyProcessor3 */, "PRO
.addProcessor("PROCESS3", MyProcessor3::new /* the ProcessorSupplier that can generate MyProcessor3 */, "PRO
// connect the state store "COUNTS" with processor "PROCESS2"
.connectProcessorAndStateStores("PROCESS2", "COUNTS");
.addSink("SINK1", "sink-topic1", "PROCESS1")
.addSink("SINK2", "sink-topic2", "PROCESS2")
.addSink("SINK3", "sink-topic3", "PROCESS3");
In the next section we present another way to build the processor topology: the Kafka Streams DSL.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
The DSL uses two main abstractions. A KStream is an abstraction of a record stream, where each data record represents a self-contained datum in the
Kafka Connect
Kafka Streams
unbounded data set. A KTable is an abstraction of a changelog stream, where each data record represents an update. More precisely, the value in a data
record is considered to be an update of the last value for the same record key, if any (if a corresponding key doesn't exist yet, the update will be considered a
create). To illustrate the difference between KStreams and KTables, lets imagine the following two data records are being sent to the stream:
PERFORMANCE
POWERED BY
1) --> ("alice", 3) . If these records a KStream and the stream processing application were to sum the values it would return 4 . If these records
were a KTable, the return would be 3 , since the last record would be considered as an update.
PROJECT INFO
ECOSYSTEM
Either a record stream (defined as KStream ) or a changelog stream (defined as KTable ) can be created as a source stream from one or more Kafka
topics (for KTable you can only create the source stream from a single topic).
CLIENTS
EVENTS
CONTACT US
APACHE
Download
@apachekafka
Windowing a stream
A stream processor may need to divide data records into time buckets, i.e. to window the stream by time. This is usually needed for join and aggregation
operations, etc. Kafka Streams currently defines the following types of windows:
Hopping time windows are windows based on time intervals. They model fixed-sized, (possibly) overlapping windows. A hopping window is defined by two
properties: the window's size and its advance interval (aka "hop"). The advance interval specifies by how much a window moves forward relative to the
previous one. For example, you can configure a hopping window with a size 5 minutes and an advance interval of 1 minute. Since hopping windows can
overlap a data record may belong to more than one such windows.
Tumbling time windows are a special case of hopping time windows and, like the latter, are windows based on time intervals. They model fixed-size, nonoverlapping, gap-less windows. A tumbling window is defined by a single property: the window's size. A tumbling window is a hopping window whose
window size is equal to its advance interval. Since tumbling windows never overlap, a data record will belong to one and only one window.
Sliding windows model a fixed-size window that slides continuously over the time axis; here, two data records are said to be included in the same window
if the difference of their timestamps is within the window size. Thus, sliding windows are not aligned to the epoch, but on the data record timestamps. In
Kafka Streams, sliding windows are used only for join operations, and can be specified through the JoinWindows class.
Joins
A join operation merges two streams based on the keys of their data records, and yields a new stream. A join over record streams usually needs to be
performed on a windowing basis because otherwise the number of records that must be maintained for performing the join may grow indefinitely. In Kafka
Streams, you may perform the following join operations:
KStream-to-KStream Joins are always windowed joins, since otherwise the memory and state required to compute the join would grow infinitely in size.
Here, a newly received record from one of the streams is joined with the other stream's records within the specified window interval to produce one result
for each matching pair based on user-provided ValueJoiner . A new KStream instance representing the result stream of the join is returned from
this operator.
KTable-to-KTable Joins are join operations designed to be consistent with the ones in relational databases. Here, both changelog streams are
materialized into local state stores first. When a new record is received from one of the streams, it is joined with the other stream's materialized state
stores to produce one result for each matching pair based on user-provided ValueJoiner. A new KTable instance representing the result stream of the
join, which is also a changelog stream of the represented table, is returned from this operator.
KStream-to-KTable Joins allow you to perform table lookups against a changelog stream ( KTable ) upon receiving a new record from another record
stream (KStream). An example use case would be to enrich a stream of user activities ( KStream ) with the latest user profile information (
Only records received from the record stream will trigger the join and produce results via ValueJoiner , not vice versa (i.e., records received from the
changelog stream will be used only to update the materialized state store). A new KStream instance representing the result stream of the join is
returned from this operator.
Depending on the operands the following join operations are supported: inner joins, outer joins and left joins. Their semantics are similar to the
corresponding operators in relational databases. a
Transform a stream
There is a list of transformation operations provided for KStream and KTable respectively. Each of these operations may generate either one or more
KStream and KTable objects and can be translated into one or more connected processors into the underlying processor topology. All these
transformation methods can be chained together to compose a complex processor topology. Since KStream and KTable are strongly typed, all these
transformation operations are defined as generics functions where users could specify the input and output data types.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
PERFORMANCE
POWERED BY
Stateless transformations, by definition, do not depend on any state for processing, and hence implementation-wise they do not require a state store
associated with the stream processor; Stateful transformations, on the other hand, require accessing an associated state for processing and producing
outputs. For example, in join and aggregate operations, a windowing state is usually used to store all the received records within the defined window
PROJECT INFO
boundary so far. The operators can then access these accumulated records in the store and compute based on them.
ECOSYSTEM
// written in Java 8+, using lambda expressions
CLIENTS
EVENTS
// initial value
// aggregating value
CONTACT US
);
KStream joined = source1.leftJoin(source2,
Download
@apachekafka
At the end of the processing, users can choose to (continuously) write the final resulted streams back to a Kafka topic through KStream.to
KTable.to .
joined.to("topic4");
If your application needs to continue reading and processing the records after they have been materialized to a topic via to above, one option is to
construct a new stream that reads from the output topic; Kafka Streams provides a convenience method called through :
// equivalent to
//
// joined.to("topic4");
// materialized = builder.stream("topic4");
KStream materialized = joined.through("topic4");
Besides defining the topology, developers will also need to configure their applications in StreamsConfig before running it. A complete list of Kafka
Streams configs can be found here.
The contents of this website are 2016 Apache Software Foundation under the terms of the Apache License v2.Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other
countries.
https://kafka.apache.org/documentation
11/4/2016
Apache Kafka
Implementation
Operations
Security
Kafka Connect
Kafka Streams
PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE
Download
@apachekafka
https://kafka.apache.org/documentation
11/4/2016