Chapter 1 - Introduction to KAFKA

Objectives
Key objectives of this chapter
 What is Microservices?
 Messaging Architectures
 What is Kafka?
 Need for Kafka
 Where is Kafka useful?
 Architecture
 Core concepts in Kafka
 Overview of ZooKeeper
 Cluster, Kafka Brokers, Producer, Consumer, Topic

1.1 Microservices
 Small, autonomous services which work well together.
 Being able to change individual components independently.
 Independent processes
 Communicate over APIs, rather than using databases directly
 High degree of autonomy
 Small, focused on doing one thing well
 A form of SOA. Typical SOA-based applications used to be monolithic.
 Microservices concept facilitates in adopting Agile Software Development.

1.2 Microservices vs Classic SOA

SOA Microservices
XML JSON
Complex to integrate Easy to integrate
Chapter 1 - Introduction to KAFKA

SOA Microservices
Heavy Lightweight
HTTP/SOAP HTTP/REST

1.3 Traditional Enterprise Application Architecture

 Classical architecture
 Typical 3 layers:
◊ client-side UI (Browser, HTML + JS)
◊ a database (RDBMS, NoSQL …)
◊ server-side application (Java, .NET, PHP, …)
 Any changes to the system involve building and deploying a new version
of the application. Changes are expensive.
 Scaling requires scaling of the entire application, rather than parts of it that
require greater resource.
 Long release cycles.

2
Chapter 1 - Introduction to KAFKA

1.4 Sample Microservices Architecture

 Applications naturally start as Monoliths, they scale and evolve to
Microservice architecture
 Applications are decomposed to components – smaller independent
service applications.
 Components are loosely coupled.

1.5 Microservices Architecture – Pros

 Multiple developers and teams can deliver relatively independently of each
other
 Can be written in different programming languages
 Can be managed by different teams
 Can use different data storage technologies
 Centralized management is minimal
 Independently deployable by fully automated deployment machinery
 Works well with Continuous Delivery
 Allows frequent releases while keeping the rest of the system available
and stable

3
Chapter 1 - Introduction to KAFKA

1.6 Messaging Architectures – What is Messaging?

 Application-to-application communication
 Supports asynchronous operations.
 Message:
 A message is a self-contained package of business data and network
routing headers.

1.7 Messaging Architectures – Steps to Messaging

 Messaging connects multiple applications in an exchange of data.
 Messaging uses an encapsulated asynchronous approach to exchange
data through a network.
 A traditional messaging system has two models of abstraction:
◊ Queue – a message channel where a single message is received
exactly by one consumer in a point-to-point message-queue pattern. If
there are no consumers available, the message is retained until a
consumer processes the message.
◊ Topic - a message feed that implements the publish-subscribe pattern
and broadcasts messages to consumers that subscribe to that topic.
 A single message is transmitted in five steps:
◊ Create
◊ Send

4
Chapter 1 - Introduction to KAFKA

◊ Deliver
◊ Receive
◊ Process

1.8 Messaging Architectures – Messaging Models

 1. Point to Point
 2. Publish and Subscribe

1.9 What is Kafka?

 In modern applications, real-time information is continuously generated by
applications (publishers/producers) and routed to other applications
(subscribers/consumers)
 Apache Kafka is an open source, distributed publish-subscribe messaging
system.
 Kafka allows integration of information of producers and consumers to
avoid any kind of rewriting of an application at either end.
 Kafka provides overcomes the challenges of real-time data usage for
consumption of data volumes that may grow in order of magnitude, larger
than the real data.

5
Chapter 1 - Introduction to KAFKA

 Kafka also supports parallel data loading in the Hadoop systems.

1.10 What is Kafka? (Contd.)

 Kafka is a unique distributed publish-subscribe messaging system written
in the Scala language with multi-language support and runs on the Java
Virtual Machine (JVM).
 Kafka relies on another service named Zookeeper – a distributed
coordination system – to function.
 Kafka has high-throughput and is built to scale-out in a distributed model
on multiple servers.
 Kafka persists messages on disk and can be used for batched
consumption as well as real-time applications.

1.11 Kafka Overview

 When used in the right way and for the right use case, Kafka has unique
attributes that make it a highly attractive option for data integration.

 Data Integration is the combination of technical and business processes

used to combine data from disparate sources into meaningful and valuable
information.
 A complete data integration solution encompasses discovery, cleansing,
monitoring, transforming and delivery of data from a variety of sources
 Messaging is a key data integration strategy employed in many distributed
environments such as the cloud.

6
Chapter 1 - Introduction to KAFKA

 Messaging supports asynchronous operations, enabling you to decouple a

process that consumes a service from the process that implements the
service.

1.12 Kafka Overview (Contd.)

1.13 Need for Kafka

 High Throughput
◊ Provides support for hundreds of thousands of messages with modest
hardware
 Scalability
◊ Highly scalable distributed systems with no downtime
 Replication

7
Chapter 1 - Introduction to KAFKA

◊ Messages can be replicated across a cluster, which provides support

for multiple subscribers and also in case of failure balances the
consumers
 Durability
◊ Provides support for persistence of messages to disk which can be
further used for batch consumption
 Stream Processing
◊ Kafka can be used along with real-time streaming applications like
spark, flink, and storm
 Data Loss
◊ Kafka with proper configurations can ensure zero data loss

1.14 Kafka Architecture

1.15 Core concepts in Kafka

 Topic

8
Chapter 1 - Introduction to KAFKA

◊ A category or feed to which messages are published

 Producer
◊ Publishes messages to Kafka Topic
 Consumer
◊ Subscribes and consumes messages from Kafka Topic
 Broker
◊ Handles hundreds of megabytes of reads and writes

1.16 Kafka Topic

 User defined category where the messages are published
 For each topic, a partition log is maintained
 Each partition basically contains an ordered, immutable sequences of
messages where each message assigned a sequential ID number called
offset
 Writes to a partition are generally sequential thereby reducing the number
of hard disk seeks
 Reading messages from partition can either be from the beginning and
also can rewind or skip to any point in a partition by supplying an offset
value

9
Chapter 1 - Introduction to KAFKA

1.17 Kafka Producer

 Application publishes messages to the topic in Kafka Cluster
 Can be of any kind like Front End, Streaming etc.
 While writing messages, it is also possible to attach a key to the message
 By attaching key the producers basically provide a guarantee that all
messages with the same key will arrive in the same partition
 Supports both async and sync modes
 Publishes as many messages as fast as the broker in a cluster can handle

10
Chapter 1 - Introduction to KAFKA

1.18 Kafka Consumer

 Application subscribes and consumes messages from brokers in Kafka
Cluster
 Can be of any kind like real-time consumers, NoSQL consumers etc.
 During consumption of messages from a topic a consumer group can be
configured with multiple consumers.
 Each consumer of consumer group reads messages from a unique subset
of partitions in each topic they subscribe to
 Messages with the same key arrive at the same consumer
 Supports both Queuing and Publish-Subscribe
 Consumers have to maintain the number of messages consumed

11
Chapter 1 - Introduction to KAFKA

1.19 Kafka Broker

 Kafka cluster basically comprised of one or more servers
 Each of the servers in the cluster is called a broker
 Handles hundreds of megabytes of writes from producers and reads from
consumers
 Retains all the published messages irrespective of whether it is consumed
or not
 If retention is configured for n days, then messages once published, it is
available for consumption for configured n days and thereafter it is
discarded

12
Chapter 1 - Introduction to KAFKA

1.20 Kafka Cluster

 A Kafka Cluster is generally fast, highly scalable messaging system
 A publish-subscribe messaging system
 Can be used effectively in place of ActiveMQ, RabbitMQ, Java Messaging
System (JMS), and Advanced Messaging Queuing Protocol (AMQP)
 Can be integrated with Hadoop Ecosystem
 Expanding of the cluster can be done with ease
 Effective for applications which involve large-scale message processing

13
Chapter 1 - Introduction to KAFKA

1.21 Why Kafka Cluster?

 Kafka is preferred in place of more traditional brokers like JMS and
AMQP?
◊ With Kafka, we can easily handle hundreds of thousands of messages
in a second, which makes Kafka a high throughput messaging system
◊ The cluster can be expanded with no downtime, making Kafka highly
scalable
◊ Messages are replicated, which provides reliability and durability
◊ Fault-tolerant

1.22 Sample Multi-Broker Cluster

14
Chapter 1 - Introduction to KAFKA

1.23 Overview of ZooKeeper

 An open source Apache project
 Provides a centralized infrastructure and services that enable
synchronization across a cluster
 Common objects used across the large cluster environments are
maintained in Zookeeper
 Objects such as configuration, hierarchical naming space etc. are
maintained in Zookeeper
 Zookeeper services are used by large scale applications to coordinate
distributed processing across large clusters

1.24 Kafka Cluster & ZooKeeper

1.25 Kafka Integration

 Databases: MongoDB/CosmosDB/CouchDB/Oracle
 Big Data: Hadoop, Spark

15
Chapter 1 - Introduction to KAFKA

 Logging: Logstash (ELK stack)

 IoT

1.26 Who Uses Kafka?

16
Chapter 1 - Introduction to KAFKA

1.27 Courses
 WA2708 – Kafka for Application Modernization
 WA2684 – Developing Microservices

1.28 Summary
 Kafka is a unique distributed publish-subscribe messaging system written
in the Scala language with multi-language support and runs on the Java
Virtual Machine (JVM).
 Kafka relies on another service named Zookeeper – a distributed
coordination system – to function.
 Kafka has high-throughput and is built to scale-out in a distributed model
on multiple servers.
 Kafka persists messages on disk and can be used for batched
consumption as well as real-time applications.

17