Apache Kafka Complete Tutorial for .NET Core & Angular Developers (Lead/Architect Level)
This tutorial is designed for Senior .NET Developers, Technical Leads, Architects, and Microservices Developers.
By the end of this guide, you'll understand:
What is Kafka?
Why Kafka was created?
Kafka Architecture
Kafka Components
Topics vs Queue
Producers & Consumers
Consumer Groups
Partitions
Replication
Offset
Broker
ZooKeeper vs KRaft
Event Driven Architecture
Integrating Kafka with .NET Core
Integrating Angular with Kafka using Web API
Real-time Microservices Example
Deployment in Docker
Azure Integration
Best Practices
Interview Questions
Chapter 1 - Why Kafka?
Imagine Amazon.
Thousands of events happen every second.
Customer places order
↓
Payment Completed
↓
Inventory Updated
↓
Invoice Generated
↓
Email Sent
↓
SMS Sent
↓
Loyalty Points Added
↓
Analytics Updated
If every service directly called another service,
Order Service
↓
Payment Service
↓
Inventory Service
↓
Email Service
↓
Notification Service
Problems
Tight Coupling
Slow
Difficult to Scale
Single Point Failure
Instead
Order Service
↓
Kafka
↓
Payment
Inventory
Shipping
Analytics
Email
Notification
Everything becomes independent.
Chapter 2 - What is Kafka?
Kafka is
A Distributed Event Streaming Platform
Think of Kafka as
Post Office
Producer
↓
Post Office (Kafka)
↓
Consumer
Producer doesn't know who receives.
Consumer doesn't know producer.
Everything is asynchronous.
Chapter 3 - Kafka Architecture
Producer
|
|
Kafka Cluster
-----------------
Broker 1
Broker 2
Broker 3
-----------------
Topic
Orders
Partition-0
Partition-1
Partition-2
|
|
Consumer Group
Kafka Components
Producer
Produces Message.
Example
Order API
Order Created
Producer sends
OrderCreated Event
Broker
Kafka Server.
Stores all messages.
Example
Broker 1
Broker 2
Broker 3
Kafka Cluster = Collection of Brokers.
Topic
Topic is a category.
Example
Orders
Payments
Inventory
Shipping
Notification
Each topic contains messages.
Partition
A Topic is divided into multiple partitions.
Example
Orders Topic
---------------------
Partition 0
Partition 1
Partition 2
Partition 3
Partitions enable
Parallel Processing
Scalability
High Throughput
Offset
Each message has an ID.
Kafka calls it Offset.
Example
Offset
0
1
2
3
4
5
Consumer remembers
Last Offset = 5
If application crashes
Restart
Continue from Offset 6.
Consumer
Reads messages.
Example
Inventory Service
Email Service
Analytics Service
Notification Service
Consumer Group
Multiple consumers work together.
Example
Consumer Group
Inventory-1
Inventory-2
Inventory-3
Kafka distributes partitions.
Example
Partition0 → Consumer1
Partition1 → Consumer2
Partition2 → Consumer3
No duplication.
Queue vs Topic
This is one of the most asked interview questions.
Traditional Queue
Producer
↓
Queue
↓
Consumer
Only ONE consumer gets message.
Example
Queue
Message
↓
Consumer A
Consumer B
Consumer C
Only Consumer A receives.
Kafka Topic
Producer
↓
Topic
↓
Consumer Group A
Consumer Group B
Consumer Group C
All groups receive same message.
Inside a group,
only one consumer receives.
Example
Order Created
↓
Orders Topic
↓
Inventory Group
↓
Email Group
↓
Analytics Group
↓
Shipping Group
Everyone gets copy.
Real Example
Customer Orders Mobile.
Order API
↓
Kafka Topic
↓
Payment
↓
Inventory
↓
Email
↓
SMS
↓
Analytics
↓
Recommendation Engine
Single Event
Many Consumers.
Queue vs Topic Comparison
| Queue | Kafka Topic |
|---|---|
| One Consumer | Multiple Consumer Groups |
| Message Removed | Message Retained |
| Point-to-Point | Publish Subscribe |
| Low Scalability | Very High |
| Low Throughput | Millions/sec |
Message Flow
Customer
↓
Angular
↓
.NET API
↓
Kafka Producer
↓
Orders Topic
↓
Broker
↓
Partition
↓
Consumer Group
↓
Inventory Service
↓
SQL
↓
Notification
↓
Email
Kafka Storage
Many developers ask
Where does Kafka save messages?
Kafka stores messages
Inside
Topic
↓
Partition
↓
Log Files
Example
orders-0.log
orders-1.log
orders-2.log
Messages are appended.
Kafka never inserts in middle.
Only append.
Offset 0
Offset 1
Offset 2
Offset 3
Kafka Message Format
Example
Key
OrderId=1001
Value
{
OrderId:1001,
Customer:"John",
Amount:2500
}
Replication
Suppose
Broker1 crashes.
Without replication
Data Lost.
With replication
Broker1
Leader
↓
Broker2
Follower
↓
Broker3
Follower
If Leader dies
Follower becomes Leader.
Producer Acknowledgement
acks=0
Fire and Forget
Fastest
Risky
acks=1
Leader confirms.
Most common.
acks=all
All replicas confirm.
Safest.
Delivery Guarantee
At Most Once
No Retry
May Lose.
At Least Once
Retry Enabled
Duplicate Possible.
Exactly Once
No Duplicate
No Loss
Used for Banking.
Kafka Ordering
Ordering guaranteed only
within a partition.
Example
Partition0
Order1
Order2
Order3
Always maintained.
Across partitions
No guarantee.
Why Partitions?
Imagine
10 Million Orders.
One partition
Single Consumer
Slow.
10 partitions
10 Consumers
10x faster.
.NET Core Integration
Architecture
Angular
↓
.NET API
↓
Kafka Producer
↓
Kafka Broker
↓
Inventory Service
↓
SQL Server
Install Package
dotnet add package Confluent.Kafka
Producer Example
using Confluent.Kafka;
var config = new ProducerConfig
{
BootstrapServers="localhost:9092"
};
using var producer =
new ProducerBuilder<string,string>(config)
.Build();
await producer.ProduceAsync(
"orders",
new Message<string,string>
{
Key="1001",
Value="{OrderId:1001}"
});
Producer sends
Topic
orders
Consumer Example
var config =
new ConsumerConfig
{
BootstrapServers="localhost:9092",
GroupId="inventory-group",
AutoOffsetReset=AutoOffsetReset.Earliest
};
using var consumer =
new ConsumerBuilder<string,string>(config)
.Build();
consumer.Subscribe("orders");
while(true)
{
var result=consumer.Consume();
Console.WriteLine(result.Message.Value);
}
ASP.NET Core Web API
Angular
POST
/api/orders
Controller
[HttpPost]
public async Task<IActionResult> Create(OrderDto order)
{
await producer.Publish(order);
return Ok();
}
Producer Service
await _producer.ProduceAsync(
"orders",
message);
Angular
Order Service
createOrder(order:any){
return this.http.post(
"/api/orders",
order);
}
Component
submit(){
this.orderService
.createOrder(this.order)
.subscribe();
}
Angular never talks directly to Kafka.
Reason
Kafka is backend infrastructure.
Angular
↓
Web API
↓
Kafka
Complete Flow
Angular
↓
Web API
↓
Kafka Producer
↓
Orders Topic
↓
Broker
↓
Partition
↓
Consumer Group
↓
Inventory Service
↓
Database
↓
Notification
↓
Email
Kafka vs RabbitMQ
| Kafka | RabbitMQ |
|---|---|
| Event Streaming | Message Queue |
| Huge Throughput | Moderate |
| Log Based | Queue Based |
| Message Retention | Message Deleted |
| Replay Possible | Difficult |
| Analytics | Task Processing |
Kafka in Microservices
Example
Customer Created
↓
customer-created topic
↓
Billing
↓
CRM
↓
Analytics
↓
Notification
↓
Search Index
No service depends on another.
Docker
version: '3'
services:
kafka:
image: bitnami/kafka
Run
docker compose up
Azure Integration
Kafka can integrate with
Azure Event Hubs (Kafka-compatible endpoint)
Azure Kubernetes Service (AKS)
Azure Container Apps
Azure Virtual Machines
Azure Monitor
Azure Key Vault (Secrets)
Azure DevOps (CI/CD)
Best Practices
Use meaningful topic names (e.g.,
orders.created.v1)Keep events immutable.
Prefer Avro or Protobuf with a Schema Registry over raw JSON for large systems.
Use keys that preserve ordering (for example,
OrderId).Avoid very large messages; store large files externally and publish references.
Configure retry, idempotent producers, and dead-letter handling where appropriate.
Monitor consumer lag and broker health.
Plan partition counts based on expected throughput and consumer parallelism.
Common Interview Questions
What is Apache Kafka?
Explain Kafka architecture.
What is a Broker?
What is a Topic?
What is a Partition?
What is an Offset?
Explain Consumer Groups.
Difference between Queue and Topic.
Kafka vs RabbitMQ.
Why are partitions needed?
How does Kafka guarantee ordering?
What happens when a broker fails?
What is replication factor?
What are producer acknowledgements (
acks)?What is consumer lag?
What is idempotent producer?
What is exactly-once processing?
What is the role of Schema Registry?
How do you scale Kafka consumers?
How do you integrate Kafka with .NET Core?
Complete E-Commerce Example
Angular UI
│
▼
ASP.NET Core Web API
│
▼
Kafka Producer Service
│
▼
┌──────────────────────────┐
│ Orders Topic │
└──────────────────────────┘
│ │ │
▼ ▼ ▼
Inventory Payment Analytics
Service Service Service
│ │ │
▼ ▼ ▼
SQL Server Email Data Warehouse
│
▼
Notification Service
A customer clicks Place Order in Angular. The Angular app calls the ASP.NET Core API. The API validates the request, stores the order (if following the Outbox pattern), and publishes an OrderCreated event to Kafka. Kafka persists the event in the orders topic, making it available to multiple consumer groups. Inventory reserves stock, Payment charges the customer, Analytics records the event, and Notification sends an email—all independently and asynchronously.
This decoupled architecture improves scalability, resilience, and maintainability because producers and consumers evolve independently.
For a production-grade implementation, the next topics to master are:
Kafka internals (segments, ISR, leader election, page cache)
KRaft architecture (ZooKeeper-free Kafka)
Schema Registry with Avro/Protobuf
Outbox Pattern with .NET and Entity Framework Core
Saga Pattern with Kafka
Retry topics and Dead Letter Topics (DLT)
Idempotent consumers and exactly-once semantics
Monitoring with Prometheus and Grafana
Deploying Kafka on Docker, Kubernetes, and Azure
End-to-end .NET 9 microservices with Angular 20 and Kafka using Clean Architecture and CQRS
These advanced topics are commonly expected in senior .NET Lead and Technical Architect interviews.
No comments:
Post a Comment