PREV NEXT

⏮ Common messaging publishing patterns  Building Data Streaming Applications with Apache Kafka Summary ⏭

Best practices
Hopefully, at this juncture, you are very well aware of Kafka Producer APIs, their internal working, and common

patterns of publishing messages to different Kafka topics. This section covers some of the best practices
associated with Kafka producers. These best practices will help you in making some of the design decisions for 
the producer component.

Let's go through some of the most common best practices to design a good producer application:

Data validation: One of the aspects that is usually forgotten while writing a producer system is to perform
basic data validation tests on data that is to be written on the Kafka cluster. Some such examples could be
conformity to schema, not null values for Key fields, and so on. By not doing data validation, you are risking
breaking downstream consumer applications and affecting the load balancing of brokers as data may not be
partitioned appropriately.

Exception handling: It is the sole responsibility of producer programs to decide on program flows with
respect to exceptions. While writing a producer application, you should define different exception classes and
as per your business requirements, decide on the actions that need to be taken. Clearly defining exceptions not
only helps you in debugging but also in proper risk mitigation. For example, if you are using Kafka for critical
applications such as fraud detection, then you should capture relevant exceptions to send e-mail alerts to the
Find answers on the fly, or master something new. Subscribe today. See pricing options.
OPS team for immediate resolution.

Number of retries: In general, there are two types of errors that you get in your producer application. The
first type are errors that producer can retry, such as network timeouts and leader not available. The second
type are errors that need to be handled by producer programs as mentioned in the preceding section.
Configuring the number of retries will help you in mitigating risks related to message losses due to Kafka
cluster errors or network errors.

Number of bootstrap URLs: You should always have more than one broker listed in your bootstrap broker
configuration of your producer program. This helps producers to adjust to failures because if one of the
brokers is not available, producers try to use all the listed brokers until it finds the one it can connect to. An
ideal scenario is that you should list all your brokers in the Kafka cluster to accommodate maximum broker
connection failures. However, in case of very large clusters, you can choose a lesser number that can
significantly represent your cluster brokers. You should be aware that the number of retries can affect your
end-to-end latency and cause duplicate messages in your Kafka queues.

Avoid poor partitioning mechanism: Partitions are a unit of parallelism in Kafka. You should always choose
an appropriate partitioning strategy to ensure that messages are distributed uniformly across all topic
partitions. Poor partitioning strategy may lead to non-uniform message distribution and you would not be able
to achieve the optimum parallelism out of your Kafka cluster. This is important in cases where you have
chosen to use keys in your messages. In case you do not define keys, then producer will use the default round-
robin mechanism to distribute your messages to partitions. If keys are available, then Kafka will hash the keys
and based on the calculated hash code, it will assign the partitions. In a nutshell, you should choose your keys
in a way that your message set uses all available partitions.

Temporary persistence of messages: For highly reliable systems, you should persist messages that are
passing through your producer applications. Persistence could be on disk or in some kind of database.
Persistence helps you replay messages in case of application failure or in case the Kafka cluster is unavailable
due to some maintenance. This again, should be decided based on enterprise application requirements. You
can have message purging techniques built in your producer applications for messages that are written to the
Kafka cluster. This is generally used in conjunction with the acknowledgement feature that is available with
Kafka Producer APIs. You should purge messages only when Kafka sends a success acknowledgement for a
message set.

Avoid adding new partitions to existing topics: You should avoid adding partitions to existing topics when
you are using key-based partitioning for message distribution. Adding new partitions would change the
calculated hash code for each key as it takes the number of partitions as one of the inputs. You would end up
having different partitions for the same key.

Support / Sign Out