Apache Kafka Interview Questions & Answers (Lead/Architect Level)
1. What is Apache Kafka?
Interview Answer
Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, scalable, and real-time data streaming. It allows applications to publish, store, and consume streams of events asynchronously.
Unlike traditional messaging systems that remove messages after they are consumed, Kafka stores messages for a configurable retention period, enabling multiple consumers to read the same events independently.
Kafka is widely used in:
Microservices
Event-Driven Architecture (EDA)
Real-time analytics
Log aggregation
IoT applications
Financial systems
E-commerce platforms
Real-Time Example
Amazon Order Processing
When a customer places an order:
Customer
↓
Order Service
↓
Kafka
↓
Payment Service
↓
Inventory Service
↓
Email Service
↓
Analytics Service
↓
Shipping Service
Every service receives the event independently.
Key Features
High Throughput
Fault Tolerance
Horizontal Scalability
Message Persistence
Event Replay
Distributed Architecture
Interview Tip
Kafka is an Event Streaming Platform, not just a Message Queue.
2. Explain Kafka Architecture
Kafka follows a distributed architecture.
Producer
│
▼
Kafka Cluster
----------------------------
Broker1
Broker2
Broker3
----------------------------
│
Orders Topic
----------------------------
Partition0
Partition1
Partition2
----------------------------
│
Consumer Group
Inventory Service
Email Service
Analytics Service
Components
Producer
Creates messages.
↓
Topic
Stores messages logically.
↓
Partition
Splits topic into multiple logs.
↓
Broker
Stores partitions.
↓
Consumer
Reads messages.
Flow
Angular
↓
.NET API
↓
Producer
↓
Topic
↓
Broker
↓
Consumer
↓
Database
Why Distributed?
If one broker crashes,
Other brokers continue serving requests.
3. What is a Broker?
A Broker is a Kafka server responsible for storing topic partitions and serving producer and consumer requests.
Example:
Kafka Cluster
Broker1
Broker2
Broker3
Each broker stores part of the data.
Responsibilities
Store messages
Handle producer requests
Serve consumers
Replicate data
Leader election
Offset management
Real Example
Topic has 6 partitions
Broker1
P0
P1
Broker2
P2
P3
Broker3
P4
P5
Interview Tip
Kafka Cluster = Collection of Brokers.
4. What is a Topic?
Topic is a logical category where messages are stored.
Example
Orders
Payments
Customers
Notifications
Producer writes
Orders Topic
Consumer reads
Orders Topic
Real Example
OrderCreated
↓
Orders Topic
Consumers
Inventory
Billing
Email
Analytics
All subscribe to the same topic.
Important
A Topic contains multiple partitions.
5. What is a Partition?
A partition is a physical subdivision of a topic.
Instead of
Orders
Kafka stores
Orders
Partition0
Partition1
Partition2
Why?
Allows parallel processing.
Suppose
1 Million Orders.
One partition
One Consumer
Slow.
Ten partitions
10 Consumers
Much faster.
Real Example
Partition0
Order1
Order4
Order7
Partition1
Order2
Order5
Order8
Partition2
Order3
Order6
Order9
Benefits
Scalability
Parallelism
Load balancing
6. What is an Offset?
Offset is the unique position of a message inside a partition.
Example
Partition0
Offset0
Offset1
Offset2
Offset3
Offset4
Consumer stores
Current Offset = 4
After restart,
Reads from Offset5.
Why Important?
Resume processing
Replay events
Fault recovery
Interview Tip
Offset is unique only within a partition, not across the entire topic.
7. Explain Consumer Groups
Consumer Group is a collection of consumers working together.
Example
Inventory Group
Consumer1
Consumer2
Consumer3
Topic
Partition0
Partition1
Partition2
Assignment
Consumer1 → Partition0
Consumer2 → Partition1
Consumer3 → Partition2
No duplicate processing occurs within the same group.
Multiple Groups
Orders Topic
↓
Inventory Group
↓
Email Group
↓
Analytics Group
Every group receives the same event independently.
Benefits
Scalability
Load balancing
Fault tolerance
8. Difference between Queue and Topic
| Queue | Kafka Topic |
|---|---|
| Point-to-Point | Publish-Subscribe |
| One consumer receives a message | Multiple consumer groups receive the same message |
| Message is typically removed after processing | Message is retained for a configured period |
| Limited replay | Replay supported |
| Suitable for task distribution | Suitable for event streaming |
Queue Example
Producer
↓
Queue
↓
Consumer A
Consumer B never receives the message.
Kafka Topic Example
Producer
↓
Orders Topic
↓
Inventory Group
↓
Email Group
↓
Analytics Group
Every group processes the event.
9. Kafka vs RabbitMQ
| Kafka | RabbitMQ |
|---|---|
| Distributed Event Streaming | Traditional Message Broker |
| Stores messages in logs | Stores messages in queues |
| Very high throughput | Moderate throughput |
| Supports replay | Replay is limited |
| Excellent for analytics and streaming | Excellent for task queues and request/response |
| Partition-based scaling | Queue-based scaling |
Real Use Cases
Kafka
Banking events
IoT
Real-time analytics
Clickstream processing
RabbitMQ
Email queues
Background jobs
Order processing tasks
RPC-style messaging
10. Why are partitions needed?
Partitions allow Kafka to scale horizontally.
Without partitions
1 Topic
↓
1 Consumer
With partitions
Partition0
Partition1
Partition2
↓
Consumer1
Consumer2
Consumer3
Benefits
Higher throughput
Better CPU utilization
Parallel processing
Horizontal scaling
11. How does Kafka guarantee ordering?
Kafka guarantees ordering only within a single partition.
Example
Partition0
Offset0 OrderCreated
Offset1 PaymentCompleted
Offset2 OrderShipped
Consumers read in offset order.
Across multiple partitions, there is no global ordering guarantee.
Best practice: If ordering is important for a business entity (for example, an order), use the same message key (OrderId) so Kafka routes all events for that key to the same partition.
12. What happens when a broker fails?
Kafka uses replication to maintain availability.
Broker1
Leader
↓
Broker2
Follower
↓
Broker3
Follower
If Broker1 fails:
Kafka elects a new leader from the in-sync replicas (ISR).
Producers and consumers continue working with the new leader.
When the failed broker comes back, it catches up and rejoins as a follower.
This minimizes downtime and prevents data loss when configured correctly.
13. What is replication factor?
The replication factor defines how many copies of each partition Kafka maintains.
Example:
Replication Factor = 3
Partition0
Leader → Broker1
Replica → Broker2
Replica → Broker3
If one broker fails, the replicas are available for leader election.
Trade-off:
Higher replication = better availability and durability.
Higher replication also requires more storage and network bandwidth.
14. What are producer acknowledgements (acks)?
The acks setting determines how many brokers must acknowledge a write before the producer considers it successful.
acks = 0
Producer does not wait.
Fastest.
Messages may be lost.
acks = 1
Waits for the leader broker only.
Good balance of speed and reliability.
Most common configuration.
acks = all (or -1)
Waits for all in-sync replicas.
Highest durability.
Slightly higher latency.
15. What is consumer lag?
Consumer lag is the difference between:
the latest offset written to a partition, and
the last offset processed by a consumer.
Example
Latest Offset = 1000
Consumer Offset = 900
Lag = 100
High lag may indicate:
Slow consumers
Insufficient consumer instances
Long-running processing
Resource bottlenecks
Monitoring consumer lag is essential in production systems.
16. What is an idempotent producer?
An idempotent producer ensures that retrying a message does not create duplicates.
Without idempotence:
Send Message
↓
Network Failure
↓
Retry
↓
Duplicate Message
With idempotence enabled:
Retry
↓
Kafka recognizes the duplicate
↓
Stores only one copy
In the .NET Confluent.Kafka client, this is enabled using:
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092",
EnableIdempotence = true
};
This is especially important in payment and financial systems.
17. What is exactly-once processing?
Exactly-once processing (EOS) ensures that each event is processed only once, even in the presence of retries or failures.
Kafka achieves this through:
Idempotent producers
Transactions
Transaction-aware consumers
Example:
Transfer ₹100
↓
Kafka Event
↓
Retry
↓
Still processed only once
This is critical in banking, inventory, and billing systems where duplicate processing is unacceptable.
18. What is the role of Schema Registry?
Schema Registry centrally manages event schemas, commonly using Avro, Protobuf, or JSON Schema.
Benefits:
Producers and consumers agree on the event structure.
Supports schema evolution while maintaining compatibility.
Reduces message size (especially with Avro).
Prevents incompatible changes.
Example:
OrderCreated
OrderId
CustomerId
Amount
If a new field is added later, compatibility rules determine whether older consumers can still process the event.
19. How do you scale Kafka consumers?
Consumers scale through:
Increasing the number of partitions.
Adding more consumer instances to the same consumer group.
Example:
Topic
6 Partitions
↓
Consumer Group
Consumer1
Consumer2
Consumer3
Consumer4
Consumer5
Consumer6
Each consumer processes one partition.
Important rules:
A consumer group cannot process more partitions concurrently than the number of partitions available.
If there are more consumers than partitions, the extra consumers remain idle.
20. How do you integrate Kafka with .NET Core?
A common architecture is:
Angular
↓
ASP.NET Core Web API
↓
Kafka Producer
↓
Orders Topic
↓
Kafka Cluster
↓
Inventory Service
↓
Payment Service
↓
Email Service
↓
Analytics Service
Step 1: Install the Kafka client
dotnet add package Confluent.Kafka
Step 2: Configure the producer
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092"
};
using var producer =
new ProducerBuilder<string, string>(config).Build();
Step 3: Publish an event
await producer.ProduceAsync(
"orders",
new Message<string, string>
{
Key = "1001",
Value = JsonSerializer.Serialize(order)
});
Step 4: Create a consumer
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "inventory-group",
AutoOffsetReset = AutoOffsetReset.Earliest
};
using var consumer =
new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
while (true)
{
var result = consumer.Consume();
Console.WriteLine(result.Message.Value);
}
Angular Integration
Angular should not connect directly to Kafka.
Instead:
Angular
↓
HTTP POST
↓
ASP.NET Core Web API
↓
Kafka Producer
↓
Kafka Topic
The Web API validates the request, persists business data if needed, and publishes the event to Kafka.
Interview Summary (30-Second Answer)
"Apache Kafka is a distributed event-streaming platform used to build scalable, fault-tolerant, and asynchronous systems. Producers publish events to topics, which are split into partitions for parallel processing. Brokers store these partitions and replicate them for high availability. Consumer groups process events independently, while offsets track processing progress. In .NET Core, we typically use the Confluent.Kafka library to publish and consume events, exposing HTTP APIs for Angular clients while Kafka handles communication between backend microservices. This architecture improves scalability, resiliency, and loose coupling compared to direct service-to-service communication."
