A microservice normally does not fail because someone forgot how to write a controller.

It fails because one service depends on another service that becomes slow, unavailable, overloaded or partially unhealthy.

That distinction is important.

Consider a very ordinary enterprise request:

Web / Mobile
     ↓
API Gateway
     ↓
Booking Service
     ↓
Pricing Service
     ↓
Inventory Service
     ↓
Payment Service

In a development environment, every service may answer in a few milliseconds.

Production is different.

One dependency starts responding in 12 seconds.

Another returns HTTP 503.

The payment provider times out.

Your application retries.

Thousands of requests start waiting.

Connection pools fill.

Threads, sockets or promises remain occupied.

More retries appear.

Eventually a small downstream problem becomes a platform-wide incident.

After working with distributed enterprise systems for many years, I treat three mechanisms as fundamental building blocks of service-to-service resilience:

1. Timeout

2. Retry

3. Circuit Breaker

They solve different problems.

Using them correctly together is far more important than simply adding a resilience library.

Key Takeaways

• Every remote call should have an intentional timeout.

• Retry only failures that have a reasonable chance of succeeding later.

• Never use immediate unlimited retries.

• Prefer exponential backoff with jitter for transient failures.

• Circuit breakers prevent applications from repeatedly calling dependencies that are likely to fail.

• Retries must be safe from a business perspective; idempotency matters.

• A retry at every layer can multiply traffic dramatically.

• Resilience policies should be based on dependency characteristics, not copied globally.

• Monitor timeout rate, retry rate, circuit state and downstream latency together.

NestJS itself documents timeouts as an important pattern for microservice calls; its microservices documentation shows use of the RxJS timeout operator to avoid waiting indefinitely for another service.

1. Start With Failure Thinking

When I review a service dependency, I do not begin with:

Which circuit-breaker package should we install?

I begin with:

What exactly happens when this dependency does not respond?

For every outbound call, I want answers to questions such as:

What is the expected latency?

What is the maximum acceptable latency?

Can the caller continue without this dependency?

Is retrying safe?

How many retries can the downstream system tolerate?

Can we return cached or degraded data?

Could duplicate execution create a business problem?

What should happen when the dependency remains unavailable?

Those answers determine the resilience policy.

2. Timeout Is the First Boundary

Imagine:

Order Service
      ↓
Inventory Service

The inventory service usually answers in:

100–300 ms

but the client has no timeout.

One network problem now allows requests to wait indefinitely or for a very long platform-default duration.

If 5,000 requests arrive while inventory is unhealthy, you can end up with thousands of outstanding calls consuming resources.

A timeout changes the contract:

Order Service
      ↓
Inventory Service
      ↓
Maximum wait: 1500 ms
      ↓
Success OR controlled failure

The important idea is:

Latency is part of the service contract.

A dependency that eventually answers after 60 seconds is frequently equivalent to a failed dependency.

NestJS's HTTP integration is based on Axios through HttpModule/HttpService, and NestJS also supports timeout handling for microservice calls.

3. A Simple NestJS Timeout

For an HTTP dependency:

import { Injectable, RequestTimeoutException } from '@nestjs/common';
import { HttpService } from '@nestjs/axios';
import { firstValueFrom, timeout, catchError } from 'rxjs';

@Injectable()
export class PricingClient {
  constructor(private readonly http: HttpService) {}

  async getPrice(productId: string) {
    return firstValueFrom(
      this.http
        .get(`/pricing/${productId}`)
        .pipe(
          timeout(1500),
          catchError((error) => {
            throw new RequestTimeoutException(
              `Pricing service unavailable for ${productId}`,
            );
          }),
        ),
    );
  }
}

The exact timeout value should not be copied blindly.

A 500 ms timeout may be sensible for an internal Redis lookup but ridiculous for a third-party payment workflow.

Timeouts should be derived from:

business SLA
+
dependency latency distribution
+
network overhead
+
remaining end-to-end request budget

4. Use a Request Budget, Not Independent Random Timeouts

Suppose the frontend expects a response in:

3 seconds

Your service performs three sequential downstream calls.

This is poor planning:

Pricing timeout:   3 sec
Inventory timeout: 3 sec
Offers timeout:    3 sec

The theoretical downstream waiting time is already nine seconds.

Instead think in terms of an overall latency budget.

Example:

Overall request budget          3000 ms
---------------------------------------
Gateway / network                200 ms
Application processing           300 ms
Pricing                          500 ms
Inventory                        700 ms
Offers                           500 ms
Safety margin                    800 ms

This forces architecture discussions around actual service-level objectives.

5. Retry Solves a Different Problem

A timeout answers:

How long am I willing to wait?

Retry answers:

Should I try again after a temporary failure?

Some failures are transient:

temporary network interruption
HTTP 502
HTTP 503
connection reset
short-lived rate limiting
brief leader election
temporary service restart

Retrying may succeed.

Other failures are not transient:

HTTP 400
invalid token
validation failure
insufficient funds
business-rule rejection
malformed payload
resource genuinely does not exist

Retrying these normally creates waste.

6. Never Retry Everything

This pattern is dangerous:

while (true) {
  try {
    return await callService();
  } catch {
    // try again forever
  }
}

It has no:

• retry limit

• backoff

• timeout budget

• classification

• circuit breaker

A failing dependency can now receive unlimited traffic from your service.

AWS's reliability guidance specifically recommends controlling retry calls, using exponential backoff, adding jitter and limiting the maximum number of retries.

7. Immediate Retry Can Become a Retry Storm

Assume:

10,000 incoming requests/min

Every request performs one downstream call.

The dependency fails.

You configure:

3 retries

Now the downstream system can receive roughly:

Original requests   10,000
Retry #1            10,000
Retry #2            10,000
Retry #3            10,000
--------------------------------
Potential calls     40,000

against a service that is already unhealthy.

If several upstream services behave the same way, the multiplication becomes much worse.

Azure's architecture guidance explicitly calls uncontrolled retry behavior a retry storm antipattern and recommends limiting attempts, applying circuit breakers and observing retry behavior.

8. Exponential Backoff

Instead of:

retry immediately
retry immediately
retry immediately

use increasing delays.

For example:

Attempt 1 → 100 ms
Attempt 2 → 200 ms
Attempt 3 → 400 ms
Attempt 4 → 800 ms

A simple calculation:

function backoff(attempt: number, baseMs = 100) {
  return baseMs * Math.pow(2, attempt);
}

But this is still incomplete.

9. Add Jitter

Imagine 1,000 service instances all fail at the same time.

If each follows exactly:

100 ms
200 ms
400 ms
800 ms

they may retry simultaneously.

That produces synchronized spikes.

Jitter introduces randomness:

function calculateDelay(
  attempt: number,
  baseMs = 100,
  maxMs = 5000,
) {
  const exponential = Math.min(
    baseMs * Math.pow(2, attempt),
    maxMs,
  );

  const jitter = Math.random() * exponential;

  return Math.floor(jitter);
}

AWS recommends jitter specifically to randomize retry intervals and reduce coordinated retry behavior.

10. A Practical Retry Helper

export async function retry<T>(
  operation: () => Promise<T>,
  options: {
    maxAttempts: number;
    baseDelayMs: number;
    shouldRetry: (error: unknown) => boolean;
  },
): Promise<T> {
  let lastError: unknown;

  for (
    let attempt = 0;
    attempt < options.maxAttempts;
    attempt++
  ) {
    try {
      return await operation();
    } catch (error) {
      lastError = error;

      if (!options.shouldRetry(error)) {
        throw error;
      }

      if (attempt === options.maxAttempts - 1) {
        break;
      }

      const exponential =
        options.baseDelayMs * Math.pow(2, attempt);

      const delay =
        Math.floor(Math.random() * exponential);

      await new Promise((resolve) =>
        setTimeout(resolve, delay),
      );
    }
  }

  throw lastError;
}

Usage:

return retry(
  () => this.inventoryClient.reserve(request),
  {
    maxAttempts: 3,
    baseDelayMs: 150,

    shouldRetry: (error) =>
      isTransientNetworkError(error) ||
      isServiceUnavailable(error),
  },
);

Notice the important part is not the loop.

It is:

shouldRetry(error)

Failure classification is part of architecture.

11. Retry Only Idempotent or Idempotency-Protected Operations

Consider:

POST /payment

Your service sends a payment request.

The payment provider processes it successfully.

The network fails before your application receives the response.

Your application sees:

timeout

Was payment made?

You do not know.

If you blindly retry:

POST /payment

you could charge the customer twice.

This is why retries and idempotency belong in the same architectural discussion.

AWS's Builders' Library explicitly discusses making APIs idempotent so retries can be performed safely.

A safer request contains:

Idempotency-Key: order-823891-payment-1

Server-side:

const existing =
  await paymentRepository.findByIdempotencyKey(
    idempotencyKey,
  );

if (existing) {
  return existing;
}

Then perform and persist the payment atomically according to the business design.

12. Circuit Breaker

Retry assumes:

The operation may succeed if I try again.

Circuit Breaker says:

The dependency is failing enough that I should temporarily stop trying.

The classical states are:

       failures
CLOSED ───────────────→ OPEN
  ▲                       │
  │                       │ cooldown
  │                       ▼
  └──────── success ─ HALF_OPEN

CLOSED

Requests flow normally.

Failures are measured.

OPEN

Calls are rejected immediately.

The failing dependency receives no additional load from this caller.

HALF_OPEN

After a recovery period, a limited number of test requests are allowed.

If they succeed:

HALF_OPEN → CLOSED

If they fail:

HALF_OPEN → OPEN

Microsoft's Azure Architecture Center describes the circuit breaker as preventing operations that are likely to fail, while retry is intended for failures that may eventually succeed.

13. Why Circuit Breaker Matters

Suppose:

Checkout Service
      ↓
Recommendation Service

Recommendations are useful but non-critical.

The recommendation service becomes unavailable.

Without a breaker:

checkout
   ↓
wait 2 sec
   ↓
retry
   ↓
wait
   ↓
retry
   ↓
finally continue

Your checkout latency may explode because of a feature that was never essential to completing an order.

With a circuit breaker:

Recommendation unhealthy
        ↓
Circuit OPEN
        ↓
Fail immediately
        ↓
Continue checkout without recommendations

This is an example of graceful degradation.

14. Not Every Dependency Deserves the Same Policy

I often see configurations like:

timeout = 5 seconds
retries = 3
breaker threshold = 50%

copied across every integration.

That is convenient.

It is rarely architecture.

Consider:

Product Recommendations

Critical? No
Fallback possible? Yes
Retry? Maybe one
Timeout? Short
Circuit breaker? Yes

Payment Authorization

Critical? Yes
Fallback possible? Usually no
Retry? Only with idempotency
Timeout? Provider-specific
Circuit breaker? Yes, carefully

Audit Event Publishing

Critical synchronously? Depends
Alternative? Queue/outbox
Retry? Asynchronous
Timeout? Short

Policies should reflect business semantics.

15. Combine Timeout + Retry + Circuit Breaker Carefully

A typical outbound dependency pipeline:

Business Service
       ↓
Circuit Breaker
       ↓
Retry Policy
       ↓
Timeout
       ↓
HTTP / RPC Client
       ↓
Dependency

Conceptually:

Request
  ↓
Is circuit open?
  ├── YES → fail fast / fallback
  │
  └── NO
       ↓
    Attempt
       ↓
    Timeout?
       ↓
Transient failure?
       ├── YES → backoff + retry
       └── NO  → fail

Be deliberate about ordering.

Otherwise a single business request can remain alive far longer than expected.

16. Calculate the Worst-Case Latency

Suppose:

timeout per attempt = 2 seconds
attempts            = 3

Ignoring backoff, worst-case waiting is already approximately:

6 seconds

Add:

backoff #1 = 250 ms
backoff #2 = 500 ms

and the total approaches:

6.75 seconds

If your endpoint SLA is:

3 seconds

the resilience policy violates the service objective by design.

This is why retry configuration must fit inside the overall request budget.

17. The Retry-at-Every-Layer Problem

Consider:

API Gateway
   ↓
Service A
   ↓
Service B
   ↓
Service C

Suppose every layer retries three times.

The multiplication can become substantial.

Conceptually:

Gateway retries A
A retries B
B retries C

A single client operation may trigger far more downstream attempts than the architect intended.

My preference is to decide where retry ownership belongs.

Do not assume every layer should retry.

18. Graceful Fallback

Circuit breakers become more valuable when the application has a fallback.

Example:

async getRecommendations(userId: string) {
  try {
    return await this.recommendationClient.get(userId);
  } catch {
    return this.popularProductsCache.get();
  }
}

Architecture:

Recommendation API
       ↓ failure
Circuit Breaker
       ↓
Fallback
       ↓
Popular Products Cache

The customer receives a useful result even though personalization is temporarily unavailable.

This is often better than converting every dependency problem into an HTTP 500.

19. What I Look at During a Production Incident

If downstream failures suddenly increase, I normally work through the problem in this order:

1. Did dependency latency increase?

2. Did timeout volume increase?

3. Did retry volume increase?

4. Which status/error classes are being retried?

5. Are multiple services retrying the same dependency?

6. Is the circuit breaker opening?

7. Are connection pools saturated?

8. Is CPU actually the bottleneck?

9. Is rate limiting occurring downstream?

10. Is the fallback path healthy?

One of the most important observations is this:

latency ↑
timeouts ↑
retries ↑
dependency traffic ↑

may indicate the application is amplifying the incident itself.

20. Production Troubleshooting Matrix

ObservationLikely issueFirst actionTimeout rate ↑dependency slow/unreachableinspect downstream latencyRetry volume ↑ rapidlytransient failure or bad retry policyclassify retried errorsCPU low, sockets/connections highblocked downstream callsshorten timeout / breakerCircuit constantly OPENsustained dependency failureinvestigate dependency/fallbackOPEN ↔ HALF_OPEN oscillationunstable dependencyreduce probes / increase recovery intervalHTTP 400 being retriedbad retry classificationstop retrying permanent errorsDB writes duplicatedunsafe retryimplement idempotencyLatency > endpoint SLAretry budget too largerecalculate total request budgetDownstream traffic spikes during outageretry stormbackoff + jitter + breaker

21. Five Mistakes I Avoid

Mistake 1 — No timeout

Every remote call waits according to infrastructure defaults.

That is not a resilience policy.

Mistake 2 — Retry every exception

Permanent errors become unnecessary traffic.

Mistake 3 — Immediate retries

You repeatedly hit the unhealthy dependency at maximum speed.

Mistake 4 — Retrying non-idempotent operations blindly

You risk duplicate business actions.

Mistake 5 — Adding circuit breakers without observability

An OPEN circuit with no metrics simply changes the failure mode from:

slow

to:

fast but unexplained

Operators need to know why the breaker opened.

22. Metrics I Expect in Production

At minimum:

dependency_request_total

dependency_latency_ms

dependency_timeout_total

dependency_retry_total

dependency_failure_total

circuit_state

circuit_open_total

fallback_total

downstream_status_code

connection_pool_usage

Dashboards should allow correlation.

For example:

Pricing latency      ↑
Pricing timeouts     ↑
Pricing retries      ↑
Circuit state        OPEN
Fallback usage       ↑

That tells a coherent story.

23. Health Checks Are Related but Different

A circuit breaker protects a caller from a dependency.

A health check helps infrastructure determine whether the service instance itself is ready or healthy.

NestJS provides Terminus integration for readiness and liveness checks, which is useful when applications run behind orchestrators such as Kubernetes.

Do not use:

circuit breaker

as a substitute for:

readiness/liveness

They solve different operational problems.

24. Production Architecture

A resilience-aware service might look like:

                       Client
                         │
                         ▼
                    API Gateway
                         │
                         ▼
                  Booking Service
                         │
               ┌─────────┴─────────┐
               │                   │
               ▼                   ▼
          Pricing Client       Inventory Client
               │                   │
          Circuit Breaker      Circuit Breaker
               │                   │
          Retry + Jitter       Retry + Jitter
               │                   │
             Timeout             Timeout
               │                   │
               ▼                   ▼
        Pricing Service      Inventory Service

               Failures
                  │
          ┌───────┴───────┐
          ▼               ▼
      Fallback         Fast Failure

                  Observability
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Metrics        Logs        Traces

This is what I mean by production resilience.

Not a library.

A coordinated architecture.

25. My Decision Framework

For every dependency I document something similar to:

dependency: pricing-service

criticality: high

timeout:
  request: 1200ms

retry:
  enabled: true
  maxAttempts: 2
  strategy: exponential-backoff
  jitter: true

retryOn:
  - connection-reset
  - http-502
  - http-503

doNotRetry:
  - http-400
  - http-401
  - http-403
  - validation-error

circuitBreaker:
  enabled: true
  failureThreshold: 50%
  minimumRequests: 20
  recoveryTimeout: 15000ms

fallback:
  enabled: false

idempotency:
  required: true

The numbers are examples—not universal settings.

What matters is that resilience behavior becomes an explicit architectural decision.

26. Final Production Checklist

Before I consider an outbound integration production-ready, I ask:

• Is there an explicit timeout?

• Does the timeout fit within the endpoint SLA?

• Are retryable failures classified?

• Is retry count finite?

• Is exponential backoff used?

• Is jitter used where appropriate?

• Are non-transient errors excluded?

• Are business operations safe to retry?

• Is idempotency implemented where necessary?

• Can retries create downstream overload?

• Should a circuit breaker protect the dependency?

• Is a fallback possible?

• Are circuit transitions observable?

• Are retries and timeouts measured?

• Are logs correlated across services?

• Have failure scenarios been load-tested?

Final Architectural Lesson

Timeout, Retry and Circuit Breaker should not be treated as three framework features.

They form part of your failure-management architecture.

A timeout prevents your application from waiting indefinitely.

A retry gives transient failures a controlled opportunity to recover.

A circuit breaker prevents an unhealthy dependency from consuming additional resources and spreading failure through the platform.

But the most important rule is:

Resilience mechanisms must reduce failure amplification—not create more of it.

When a downstream dependency is struggling, the correct architecture should reduce pressure, bound waiting time, preserve critical operations and give the system room to recover.

That is what separates a microservice that works in a test environment from one that can survive real production conditions.

Related Architecture Guides

• Kafka Backpressure in Production NestJS Systems

• Next: Kafka Idempotency and Duplicate Event Handling

• Coming next: Transactional Outbox Pattern with Kafka

• Spring Boot vs NestJS for Enterprise Microservices

• Developer to Solution Architect: A Practical Roadmap