🧪 Practical Example — Implementing Resilience in .NET Core Using Polly

 🏗️ Scenario

Let’s say we have a microservice called OrderService that needs to call another service — PaymentService — through an HTTP API.
We want to make sure our OrderService:

• Retries if the call fails (Retry Pattern)

• Stops repeated failing calls (Circuit Breaker Pattern)

• Times out long requests (Timeout Pattern)

• Returns fallback data if all else fails (Fallback Pattern)

---

🧰 Step 1: Install Required Package

Run this command in your OrderService project:

dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly

---

⚙️ Step 2: Configure Resilience Policies in Program.cs

using Polly;
using Polly.Extensions.Http;

var builder = WebApplication.CreateBuilder(args);

// Define resilience policies
var retryPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()
    .WaitAndRetryAsync(3, retryAttempt =>
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))); // exponential backoff

var circuitBreakerPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()
    .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)); // break after 5 failures

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(3); // 3-second timeout

// Add HttpClient with resilience policies
builder.Services.AddHttpClient("PaymentServiceClient", client =>
{
    client.BaseAddress = new Uri("https://payments.example.com/api/");
})
.AddPolicyHandler(retryPolicy)
.AddPolicyHandler(circuitBreakerPolicy)
.AddPolicyHandler(timeoutPolicy);

builder.Services.AddControllers();

var app = builder.Build();
app.MapControllers();
app.Run();

---

💡 Step 3: Use HttpClient in Controller

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class OrderController : ControllerBase
{
    private readonly HttpClient _httpClient;

    public OrderController(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("PaymentServiceClient");
    }

    [HttpPost("create")]
    public async Task<IActionResult> CreateOrder([FromBody] object order)
    {
        try
        {
            var response = await _httpClient.PostAsJsonAsync("process", order);
            
            if (response.IsSuccessStatusCode)
                return Ok("Order placed successfully!");
            
            return StatusCode((int)response.StatusCode, "Payment service failed.");
        }
        catch (BrokenCircuitException)
        {
            // Circuit breaker is open, fallback response
            return StatusCode(503, "Payment service temporarily unavailable. Try again later.");
        }
        catch (Exception ex)
        {
            // Fallback or logging
            return StatusCode(500, $"Unexpected error: {ex.Message}");
        }
    }
}

---

🔍 Step 4: Add Health Check for Kubernetes or Azure

Add in Program.cs:

builder.Services.AddHealthChecks();
app.MapHealthChecks("/health");

✅ This allows Kubernetes or Azure App Service to detect and restart unhealthy instances automatically.

---

📊 Step 5: Add Observability (Optional)

Integrate Application Insights, Serilog, or ELK Stack to monitor:

• Retry attempts

• Circuit breaker states

• Timeout events

This helps detect performance or availability issues early.

---

🧠 Output Behavior Summary

Failure Type	Pattern Triggered	System Reaction
Temporary network error	Retry	Automatically retries
Persistent downstream failure	Circuit Breaker	Blocks further calls temporarily
Long response	Timeout	Cancels request after 3 seconds
Service completely down	Fallback	Returns cached/default message
Pod unhealthy	Health Check	Auto restarted by Kubernetes

---

🚀 Result

With these configurations:

• Your microservices become self-healing.

• You prevent cascading failures.

• The system delivers consistent performance and high availability.

✅ Pro Tip: Combine Polly with Serilog for logging and OpenTelemetry for tracing to create a fully observable resilient microservice system.

🧱 How to Handle Resilience in Microservices — Patterns, Tools, and Best Practices

 🌐 Introduction

In a microservices architecture, dozens of services talk to each other over the network. What happens if one service fails or becomes slow?
That’s where resilience comes in.

Resilience means designing systems that can recover gracefully from failures and keep working smoothly even when some parts break. It’s not about avoiding failure — it’s about surviving it.

Let’s explore how to achieve this using the best resilience patterns, tools, and real-world examples.

---

⚠️ Why Resilience Matters in Microservices

Since microservices depend on each other through APIs or message queues, a failure in one can trigger a chain reaction across the system. Common failure causes include:

• Network latency or packet loss

• API or database downtime

• Slow response times

• Memory leaks or thread exhaustion

• Sudden traffic spikes

Without resilience, such issues can cause cascading failures that bring down the entire application.

---

🧰 Key Resilience Patterns in Microservices

Here are the most popular design patterns that help microservices withstand failures:

---

1. 🔁 Retry Pattern

Automatically retries a failed operation after a short delay — useful for temporary errors like network glitches.

Example:
If the Payment API fails once, the system retries 2–3 times before giving up.

C# Example using Polly:

var policy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetry(3, retry => TimeSpan.FromSeconds(Math.Pow(2, retry)));

await policy.ExecuteAsync(() => httpClient.GetAsync("https://payments/api"));

✅ Best for: Temporary, recoverable failures such as timeouts or transient errors.

---

2. ⚡ Circuit Breaker Pattern

Prevents cascading failures by stopping calls to an unresponsive service for a specific time.

Example:
If InventoryService fails multiple times, the circuit opens and blocks further calls for 30 seconds.

Polly Example:

var breaker = Policy
    .Handle<HttpRequestException>()
    .CircuitBreaker(5, TimeSpan.FromSeconds(30));

✅ Best for: Avoiding system overload when dependencies are unstable.

---

3. 🧩 Bulkhead Pattern

Isolates resources (like thread pools or memory) between services or components — just like watertight compartments on a ship.

Example:
If the Reporting module gets overloaded, it won’t affect the Order or Payment services.

✅ Best for: Multi-threaded, high-traffic systems.

---

4. ⏳ Timeout Pattern

Defines how long to wait before giving up on a request.

Example:
If a service call doesn’t respond within 2 seconds, abort it and trigger a fallback.

Polly Example:

var timeoutPolicy = Policy.TimeoutAsync(2); // seconds

✅ Best for: Preventing blocked threads and long response times.

---

5. 🔄 Fallback Pattern

Provides a default or cached response when a dependent service is unavailable.

Example:
If the Recommendation Service is down, show cached recommendations instead.

✅ Best for: Maintaining a smooth user experience during outages.

---

6. 🚦 Rate Limiting & Throttling

Limits the number of requests a service can process in a given timeframe.

Example:
Allow only 100 API requests per second per user to prevent overload.

✅ Best for: Protecting services from spikes or denial-of-service attacks.

---

7. ⚖️ Load Balancing

Distributes incoming requests evenly across multiple instances of a service.

Common Tools:
Azure Front Door, AWS Elastic Load Balancer (ELB), Nginx, or Kubernetes Services.

✅ Best for: Achieving scalability and high availability.

---

8. 🌤️ Graceful Degradation

Reduces functionality instead of complete failure.

Example:
If premium analytics is down, show basic reporting features.

✅ Best for: Preserving usability during partial outages.

---

🧩 Tools and Frameworks for Resilience

Platform	Tool / Library	Purpose
.NET Core	Polly	Retry, Circuit Breaker, Timeout, Fallback
Java	Resilience4j / Hystrix	Fault tolerance & resilience
Kubernetes	Liveness & Readiness Probes	Auto-heal unhealthy pods
API Gateway	Rate Limiting, Throttling	Traffic management
Azure / AWS	Front Door, Load Balancer	Failover and routing

---

📊 Observability for Resilience

Resilience without monitoring is like driving blindfolded. Use these tools to track and react to issues:

• Logging: Serilog, ELK Stack

• Metrics: Prometheus, Azure Monitor

• Tracing: Jaeger, Zipkin, Application Insights

These tools help identify failure patterns, retry loops, or slow dependencies in real time.

---

🏗️ Real-World Architecture Example

Client → API Gateway → OrderService
                        ↘ InventoryService → Database
                        ↘ PaymentService → External Gateway

Each service:

• Implements Retry & Timeout (Polly)

• Uses Circuit Breaker for external dependencies

• Falls back to cache or default data when needed

• Is monitored with health checks in Kubernetes

---

🧭 Summary of Resilience Patterns

Pattern	Purpose	Example Tool
Retry	Handle transient failures	Polly
Circuit Breaker	Stop cascading failures	Polly / Resilience4j
Timeout	Prevent blocked threads	Polly
Bulkhead	Isolate resources	Thread Pools
Fallback	Provide default response	Polly
Rate Limiting	Control request load	API Gateway
Health Checks	Detect service health	ASP.NET Core / Kubernetes

---

🚀 Conclusion

Resilience is not an afterthought — it’s the foundation of a reliable microservices system.
By applying the right resilience patterns, monitoring, and tools, you can ensure your application stays stable even when parts of it fail.

🗣️ “Failures are inevitable — resilience makes them invisible to your users.”