1. What is Kafka?
Apache Kafka is a popular distributed event streaming platform.
Kafka acts as a middle layer between applications or services that produce data and applications or services that consume data.
Kafka helps decouple producers and consumers, so they do not need to communicate with each other directly.
The basic flow is:
Producer → Kafka → Consumer
There can be multiple producers and multiple consumers, and they can operate independently.
2. Producers
Producers are applications, services, or APIs that create and send events to Kafka.
Producers use the Kafka Producer API to publish events to Kafka topics.
For example, an application might generate an event such as:
"Customer placed an order."
The producer sends this event to a Kafka topic.
Examples of Kafka Producers
- E-commerce application Customer places an order. The Order Service sends an "Order Created" event to Kafka. Producer: Order Service Topic: OrderEvents
- Banking application A customer makes a payment. The Payment Service sends a "Payment Completed" event. Producer: Payment Service Topic: PaymentEvents
- Healthcare application A patient schedules an appointment. The Appointment Service sends a "Appointment Created" event. Producer: Appointment Service Topic: AppointmentEvents
- Mobile application A user logs into a mobile app. The application can send a "User Login" event. Producer: Mobile application/backend service Topic: UserActivity
- IoT device A temperature sensor continuously sends temperature readings. The device or IoT gateway sends those readings to Kafka. Producer: IoT device/gateway Topic: TemperatureEvents
- Website A user clicks a product or searches for something. The website/backend can send a user activity event to Kafka. Producer: Website/backend service Topic: UserActivity
The flow will be:
Producer → sends event → Kafka Topic
For example:
Order Service → "Order Created" → OrderEvents Topic
3. Topics
Kafka organizes events into different topics based on different criteria.
A topic is a named stream or logical category where related events are published.
For example:
- CustomerEvents
- OrderEvents
- PaymentEvents
Multiple producers can publish events to the same topic.
For example, multiple applications may publish different order-related events to the OrderEvents topic.
4. Partitions
A Kafka topic can have multiple partitions.
For example:
OrderEvents Topic
→ Partition 0 → Partition 1 → Partition 2
Partitions allow Kafka to scale and process events in parallel.
Events within a partition are stored in an ordered sequence.
One important point is:
Ordering is guaranteed within a partition, not across the entire topic.
For example, if events are distributed across three partitions, Kafka does not guarantee the overall ordering of events across all three partitions.
5. Offsets
Each record in a Kafka partition has an offset.
An offset is a position number that identifies a record's location within a partition.
For example:
Partition 0
Offset 0 → Offset 1 → Offset 2 → Offset 3 → Offset 4
Consumers use offsets to keep track of their position while reading records.
For example, if a consumer has processed records through offset 3 and has committed that offset, it can continue from the next record, offset 4, when it resumes.
So, offsets help consumers keep track of where they are in a partition.
6. Brokers and Kafka Clusters
A Kafka broker is a Kafka server that stores and serves Kafka data.
A Kafka environment can have multiple brokers working together.
A group of Kafka brokers working together is called a Kafka cluster.
For example:
Kafka Cluster
→ Broker 1 → Broker 2 → Broker 3
Kafka clusters provide scalability, availability, and fault tolerance.
Broker, Cluster, Topic, Partition, and Offset
- Broker = A Kafka server.
- Cluster = A group of Kafka brokers working together.
- Topic = A named stream or logical category of related events.
- Partition = A topic is divided into partitions for scalability and parallel processing.
- Offset = A position number that identifies a record within a partition.
A broker can store partitions from multiple topics.
For example:
Broker 1 might have:
- 2 partitions from OrderEvents
- 3 partitions from CustomerEvents
- 1 partition from PaymentEvents
Broker 2 might have:
- 1 partition from OrderEvents
- 1 partition from CustomerEvents
- 2 partitions from PaymentEvents
Broker 3 might have:
- 2 partitions from OrderEvents
A broker can store an entire topic (all partitions of a topic), or it can store partitions from multiple topics.
For example, if OrderEvents has three partitions, all three partitions could be stored on the same broker. Alternatively, those partitions can be distributed across multiple brokers.
So, a broker can store partitions from multiple topics, and the partitions of a topic can be distributed across multiple brokers.
This distribution allows Kafka to scale across multiple brokers.
7. Consumers
Consumers are applications or services that read events from Kafka topics.
For example, a consumer might:
- Update a database
- Send a notification
- Call another API
- Perform business processing
- Send a processed event to another Kafka topic
Consumers use the Kafka Consumer API to read events from Kafka topics.
8. Consumer Groups
Multiple consumers can work together as a consumer group.
For example:
OrderEvents Topic
→ Partition 0 → Consumer 1 → Partition 1 → Consumer 2 → Partition 2 → Consumer 3
This allows multiple consumers to process different partitions in parallel.
Within a consumer group, a partition is normally assigned to only one consumer at a time.
However, one consumer can be assigned multiple partitions.
This allows the workload to be distributed among consumers.
9. Batch Data vs. Streaming Data
Batch Data
Batch Data: Data is collected and stored over a period of time and then processed together as a group, such as processing a day's transactions at the end of the day.
The important point is that batch processing does not necessarily involve Kafka at all.
For example, suppose a company wants to process all customer orders once every night.
During the day:
Customer orders → Application → Database / Data Lake / Files
The data is accumulated there throughout the day.
At the end of the day:
Database / Data Lake / Files → Batch Processing Job → Reports / Data Warehouse / Other Systems
Streaming Data
Streaming Data: Data is processed continuously as it is generated.
Kafka is commonly used for streaming data, allowing applications to publish and consume events in near real time.
Where does Kafka fit in Batch Data or Streaming Data?
Kafka is primarily designed for streaming/event-driven data.
Data can be sent to Kafka as soon as events are generated:
Customer places an order
↓
Producer
↓
Kafka Topic
↓
Consumer processes the event
Kafka does store the events, but it is not a place where you wait until the end of the day and then process everything.
Consumers can read and process events continuously as they become available.
Note: Kafka can also be used in batch-like processing patterns because consumers can read and process many records together. However, Kafka's main strength is continuous event streaming.
10. Kafka in Simple Terms
The basic flow can be summarized as:
Producers → Kafka Topics → Partitions → Consumers
Producers do not need to know who the consumers are.
Consumers do not need to know who produced the events.
Kafka acts as the middle layer that receives, stores, and makes events available to consumers.
In one simple example:
Customer places an order
↓
Producer sends "Order Created" event
↓
Kafka receives the event and stores it in an OrderEvents topic
↓
The topic's partition stores the event with an offset
↓
Consumer reads the event
↓
Consumer updates the database, sends a notification, or performs other business processing
This is one of the fundamental ways Kafka is used in modern distributed applications.
#kafka