Bulkhead Pattern: Isolating Supplier, Payment and Database Failures

The idea

A ship uses bulkheads so one flooded compartment does not sink the entire vessel.

In software, the limited resources may be:

• threads;
• async concurrency;
• sockets;
• HTTP connection pools;
• database connections;
• worker processes;
• queue capacity.

Travel aggregation is a good example.

Supplier A becomes slow

If all suppliers use the same unlimited concurrency, Supplier A can consume nearly every available connection while Supplier B and C are healthy.

Dependency-specific limits

Flight Supplier A -> concurrency 10
Flight Supplier B -> concurrency 20
Hotel Supplier    -> concurrency 15
Payment Provider  -> concurrency 8

Node:

import pLimit from 'p-limit';

const supplierALimit = pLimit(10);
const supplierBLimit = pLimit(20);
const paymentLimit   = pLimit(8);

function callSupplierA(req: SearchRequest) {
  return supplierALimit(
    () => supplierA.search(req),
  );
}

Resilience4j Bulkhead

resilience4j:
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 8
        maxWaitDuration: 50ms

      supplierA:
        maxConcurrentCalls: 12
        maxWaitDuration: 0ms

I often prefer quick rejection over waiting five seconds in a local queue when the downstream is already saturated.

Bulkhead + breaker

request
  |
bulkhead limit
  |
circuit breaker
  |
timeout
  |
dependency

The bulkhead protects caller resources.

The breaker decides whether calls should be attempted based on dependency health.

The timeout bounds one attempt.

These are complementary.

Hidden shared bottleneck

Suppose I create different HTTP pools but all flows write to the same database with 20 connections.

The DB remains the common failure domain.

Bulkhead design starts by identifying the real constrained resource, not just drawing separate boxes.

Queue length

A semaphore with an unlimited waiting queue is not a complete bulkhead.

I bound both:

active = 10
waiting <= 20
beyond -> reject / degrade

Otherwise overload moves from active work to memory.

Production metrics

• active calls per bulkhead;
• waiting calls;
• rejected calls;
• dependency latency;
• timeout rate;
• breaker state;
• overall request degradation.

Failure scenarios

Too-small bulkhead

Healthy requests are rejected. Tune from observed concurrency/latency.

Too-large bulkhead

It stops protecting the downstream and becomes equivalent to no limit.

Every pod has 20 calls but 100 pods exist

Cluster-level pressure is 2,000 calls. Instance-local limits must be considered with replica count.

Autoscaling multiplies downstream load

HPA can create more callers faster than a fixed supplier/database can scale. Autoscaling and bulkhead policy must be coordinated.

Production checklist

• Identify real constrained resource.
• Set max active work.
• Bound wait queue.
• Consider replica count.
• Monitor rejection.
• Pair with timeout.
• Pair with circuit breaker.
• Define degraded behavior.
• Load-test a slow dependency.
• Re-evaluate after autoscaling changes.

FAQ

Bulkhead vs circuit breaker?

Bulkhead controls resource allocation. Circuit breaker reacts to dependency health.

Can Kubernetes replicas replace a bulkhead?

No. More replicas can multiply pressure.

Where do I implement bulkheads in Node?

Semaphores/concurrency limiters, worker pools, HTTP agent limits or dedicated queues.

Related architecture guides

• Circuit Breaker in Node

• Backpressure in Kafka

  Architect's final take

  Bulkheads are one of the simplest ways to reduce blast radius. I prefer a controlled partial degradation to letting one dependency consume the complete service.