1) Service boundaries &
contracts
- One business capability per
service
(e.g., Orders, Payments, Catalog). Each owns its own database.
- API surface: REST (JSON) or gRPC for
synchronous calls; events (Kafka/RabbitMQ/Azure Service Bus) for
async integration.
- Version your APIs (e.g., /v1), keep backward compatible
changes, use OpenAPI/Swagger and contract tests (Pact).
2) Resilience patterns
(timeouts, retries, circuit breaker, bulkhead)
Use HttpClientFactory
+ Polly on every outbound call:
builder.Services.AddHttpClient("CatalogClient",
c =>
{
c.BaseAddress = new Uri(config["CatalogUrl"]);
c.Timeout
= TimeSpan.FromSeconds(3);
})
.AddTransientHttpErrorPolicy(p =>
p.WaitAndRetryAsync(
retryCount: 3,
sleepDurationProvider: attempt => TimeSpan.FromMilliseconds(200 *
Math.Pow(2, attempt)), // expo backoff
onRetry:
(outcome, ts, attempt, ctx) =>
logger.LogWarning(outcome.Exception,
"Retry {Attempt}", attempt)))
.AddTransientHttpErrorPolicy(p =>
p.CircuitBreakerAsync(
handledEventsAllowedBeforeBreaking: 5,
durationOfBreak: TimeSpan.FromSeconds(30)))
.AddPolicyHandler(Policy.BulkheadAsync<HttpResponseMessage>(maxParallelization:
50, maxQueuingActions: 100));
Rules of
thumb
- Always set timeouts
on I/O.
- Prefer idempotent
operations; send Idempotency-Key headers for POSTs where possible.
- Use cancellation tokens
end-to-end.
3) Circuit breaker semantics
- Trip when a dependency is
unhealthy; fail fast and return a cached/partial response or
friendly error.
- Expose breaker state via metrics
and health checks (see §5).
4) Data & database
handling (consistency without distributed transactions)
- Database per service (EF Core or Dapper). No
cross-service joins.
- Propagate data between
services
using domain events—not shared tables.
- Ensure atomicity with
the Outbox pattern:
- Write your entity and an OutboxMessage
row in the same DB transaction.
- A background Outbox
Publisher reads unpublished rows and emits them to the broker, then
marks them sent.
// Inside your OrderService command handler
await using var tx =
db.Database.BeginTransactionAsync();
db.Orders.Add(order);
db.Outbox.Add(new OutboxMessage {
Type =
"OrderPlaced",
Payload =
JsonSerializer.Serialize(new { order.Id, order.Total, order.CustomerId }),
OccurredOn = DateTimeOffset.UtcNow
});
await db.SaveChangesAsync();
await tx.CommitAsync();
// BackgroundService publisher
protected override async Task
ExecuteAsync(CancellationToken ct)
{
while
(!ct.IsCancellationRequested)
{
var
messages = await db.Outbox
.Where(m => m.SentOn == null)
.OrderBy(m => m.OccurredOn)
.Take(100).ToListAsync(ct);
foreach (var msg in messages)
{
await broker.PublishAsync(msg.Type, msg.Payload, ct);
msg.SentOn = DateTimeOffset.UtcNow;
}
await
db.SaveChangesAsync(ct);
await
Task.Delay(TimeSpan.FromSeconds(1), ct);
}
}
- Updating “relative/related”
databases:
consume those events in other services to update local read models
(CQRS). Use Sagas (orchestration or choreography) for multi-step
processes (e.g., Order → Reserve Inventory → Process Payment → Arrange
Shipping), with timeouts + compensation.
5) Health checks,
readiness, and monitoring
Add liveness
(process alive) and readiness (deps OK) endpoints:
builder.Services.AddHealthChecks()
.AddSqlServer(config.GetConnectionString("OrdersDb"))
.AddRabbitMQ(config["RabbitMq:Connection"])
.AddUrlGroup(new Uri(config["CatalogUrl"] +
"/health"), name: "catalog");
app.MapHealthChecks("/health/live", new
HealthCheckOptions { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new
HealthCheckOptions {
Predicate
= _ => true,
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
- Kubernetes/Containers probe /health/live and gate traffic on /health/ready.
- Add self-diagnostics
(thread pool starvation, GC info) via EventCounters exported by
OpenTelemetry.
6) Observability (logs,
traces, metrics)
Unified,
structured telemetry makes
bugs easy to fix.
OpenTelemetry (logs + traces + metrics) →
backends like Grafana Tempo/Prometheus/Loki, Jaeger, Elastic,
or Azure Application Insights.
builder.Services.AddOpenTelemetry()
.WithTracing(t => t
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSqlClientInstrumentation()
.AddSource("OrderService")
.AddOtlpExporter()) // or Azure Monitor exporter
.WithMetrics(m => m
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddHttpClientInstrumentation()
.AddPrometheusExporter());
app.MapPrometheusScrapingEndpoint(); // /metrics
Structured
logging (e.g.,
Serilog → console/JSON + sink of choice):
Log.Logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.Enrich.WithCorrelationId()
.WriteTo.Console(new JsonFormatter())
.WriteTo.Seq(config["SeqUrl"])
// or Elastic/Azure
.CreateLogger();
builder.Host.UseSerilog();
- Include CorrelationId/TraceId
in every log; propagate with middleware:
app.Use(async (ctx, next) =>
{
var cid =
ctx.Request.Headers["X-Correlation-ID"].FirstOrDefault() ??
Guid.NewGuid().ToString();
ctx.Items["CorrelationId"] = cid;
ctx.Response.Headers["X-Correlation-ID"] = cid;
using
(LogContext.PushProperty("CorrelationId", cid))
await
next();
});
- Emit domain events
and key business metrics (orders placed, failures, latencies, queue
depth).
7) API gateway / proxy
servers
- Use a Gateway for
routing, TLS, auth/ratelimiting, request shaping:
- YARP (in-process .NET reverse
proxy) or Ocelot; at the edge you can also use NGINX/Envoy.
- Centralize AuthN/Z
(OAuth2/OIDC via Azure AD, Auth0, or Duende IdentityServer), rate
limits, request/response size limits, CORS, and header
normalization at the gateway.
- Prefer service discovery
(K8s DNS) and mTLS in cluster; consider a service mesh
(Istio/Linkerd) for uniform retries/mtls/traffic shifting and better
telemetry without code changes.
8) Security &
configuration
- Never share DBs across services.
Grant least privilege per service.
- Secrets via Azure Key
Vault / K8s secrets; bind with IOptions<T> and validate on startup.
- Enforce TLS everywhere,
JWT validation, rotating keys, and input validation
(FluentValidation).
9) Deployment & runtime
- Containerize each service (Docker),
build once, promote the same image through Dev → QA → Prod.
- Orchestrate with Kubernetes
(AKS) or Azure App Services for simpler cases.
- Blue/Green or Canary deployments; mesh/gateway
can split traffic gradually.
- Horizontal scaling via HPA
(CPU/RPS/latency). Keep services stateless; externalize state
(DB/cache).
- Caching: local memory for tiny TTLs,
Redis for shared cache; always set cache keys + expirations.
10) CI/CD and quality gates
- GitHub Actions / Azure
DevOps Pipelines:
- Build →
unit/integration/contract tests → SCA & SAST → container scan →
publish image → deploy via Helm/Bicep/Terraform.
- Block merges without green tests
and security checks.
- Spin ephemeral test
environments per PR when feasible.
11) Diagnostics & “easy
bug fixing”
- Global exception handler →
consistent error envelopes with traceId:
app.UseExceptionHandler(builder =>
{
builder.Run(async ctx =>
{
var
traceId = Activity.Current?.Id ?? ctx.TraceIdentifier;
var
problem = new { traceId, message = "Unexpected error." };
ctx.Response.StatusCode = 500;
await
ctx.Response.WriteAsJsonAsync(problem);
});
});
- Feature flags (Azure App Configuration)
for safe rollouts; dark-launch code paths.
- Replayable messages (don’t auto-ack until
processed). Keep dead-letter queues and dashboards.
- Synthetic checks (probes that place a tiny
test order in lower envs and alert on end-to-end failure).
12) Database migrations
& uptime
- Apply EF Core migrations
at startup (carefully) or via migration jobs;
- Prefer expand-and-contract:
add new columns/tables (expand), release code that writes both, backfill,
then remove old fields (contract).
- For read models, permit rebuild
from event streams or upstream APIs if corrupted.
13) Monitoring & alerts
you’ll actually use
- Golden signals per service: Latency,
Traffic, Errors, Saturation.
- Alert on SLIs/SLOs
(e.g., p95
< 300ms, error rate < 1%), queue lag, breaker open
rate, DB CPU, connection pool exhaustion.
- Route alerts to Teams/Slack
with runbooks linked (how to diagnose/rollback).
14) Local/dev ergonomics
- Docker-compose for local broker + DB +
dependencies.
- Seed data + test fixtures;
make services start with sensible defaults.
- Makefile/PS scripts for common tasks;
one-command bootstrap.
Minimal reference architecture (quick checklist)
- ASP.NET Core services (+
gRPC if needed)
- HttpClientFactory + Polly
(timeouts/retries/circuit breaker/bulkhead)
- Outbox +
BackgroundPublisher; events over Kafka/RabbitMQ/Azure Service Bus
- DB per service (EF
Core/Dapper) + migrations (expand/contract)
- OpenTelemetry (traces/metrics/logs)
→ Prometheus/Grafana/Tempo or App Insights
- Serilog structured logging +
correlation IDs
- Health checks: /health/live, /health/ready
- API Gateway: YARP/Ocelot;
edge NGINX/Envoy; OAuth2/OIDC
- Containerized; AKS/App
Service; blue/green or canary
- CI/CD with tests, scans,
infrastructure as code
- Feature flags; DLQs;
synthetic probes; runbooks
