PostgreSQL High Availability: Streaming Replication, WAL, Backup and Recovery

A production approach to PostgreSQL replicas, promotion, read consistency, WAL retention, base backups and tested point-in-time recovery.

Why this matters in production

A PostgreSQL standby improves availability, but reliable recovery still depends on WAL, backup retention, promotion discipline and application behavior during role changes.

The database is not a passive persistence layer. It is a concurrency system, a cache hierarchy, a durability mechanism and often the most stateful dependency in the architecture. I therefore review database design together with API behavior, background jobs, failure recovery, deployment and observability.

A design that performs well on a development dataset can fail very differently under production cardinality. More rows or documents change selectivity, working-set size, lock duration, cache hit rate, replication lag and maintenance cost. My objective is predictable behavior rather than one impressive benchmark.

The decision model I use

1. Define synchronous versus asynchronous needs

Synchronous replication can reduce data-loss exposure but adds write-latency and availability trade-offs.

In an architecture review I convert this into a measurable question: what workload assumption makes this choice correct, what signal would tell us that assumption is no longer true, and what is the operational response? That prevents a database feature from becoming a permanent design decision simply because it worked on the first release.

2. Monitor replay lag

A replica can be connected yet too far behind for safe promotion or correctness-sensitive reads.

In an architecture review I convert this into a measurable question: what workload assumption makes this choice correct, what signal would tell us that assumption is no longer true, and what is the operational response? That prevents a database feature from becoming a permanent design decision simply because it worked on the first release.

3. Fence the old primary during failover

Split-brain prevention matters more than a fast promotion.

In an architecture review I convert this into a measurable question: what workload assumption makes this choice correct, what signal would tell us that assumption is no longer true, and what is the operational response? That prevents a database feature from becoming a permanent design decision simply because it worked on the first release.

4. Preserve WAL for the recovery objective

Point-in-time recovery requires a base backup plus a complete enough WAL archive.

In an architecture review I convert this into a measurable question: what workload assumption makes this choice correct, what signal would tell us that assumption is no longer true, and what is the operational response? That prevents a database feature from becoming a permanent design decision simply because it worked on the first release.

5. Use connection routing that understands roles

Applications should reconnect to the promoted primary without manual endpoint edits where practical.

In an architecture review I convert this into a measurable question: what workload assumption makes this choice correct, what signal would tell us that assumption is no longer true, and what is the operational response? That prevents a database feature from becoming a permanent design decision simply because it worked on the first release.

6. Practice recovery to a new environment

A recovery test should prove data, application connectivity and the measured time to restore service.

In an architecture review I convert this into a measurable question: what workload assumption makes this choice correct, what signal would tell us that assumption is no longer true, and what is the operational response? That prevents a database feature from becoming a permanent design decision simply because it worked on the first release.

Reference implementation

# Example operational concepts (paths/topology vary)
pg_basebackup -h primary-db -D /backup/base -Fp -Xs -P

# Recovery relies on archived WAL and a tested restore procedure.
# Promotion should be coordinated with fencing and application routing.
pg_ctl promote -D /var/lib/postgresql/data

The example is deliberately focused on the decision rather than framework boilerplate. In production I also capture the query or command frequency, expected cardinality, latency target and failure behavior so the database choice can be tested against an explicit workload.

Data modeling and ownership

I want every table, collection or document family to have a clear application owner. Shared read access may be appropriate, but shared write ownership creates coupling quickly. When multiple services write the same data directly, schema changes become coordinated releases and business invariants become difficult to locate.

I also distinguish transactional data from analytical or historical data. A primary operational database should not carry unlimited reporting pressure just because the information is available there. Read replicas, projections, warehouses, archival stores or asynchronous exports can protect the transactional path.

Performance methodology

I do not begin tuning with configuration switches. I start with a representative slow operation and evidence: execution plan, rows/keys examined, buffer/cache behavior, lock waits, I/O, CPU, connection saturation and the distribution of latency.

The first optimization is frequently reducing work: read fewer rows, project fewer columns/fields, index the actual predicate, remove a query loop, batch work, or change the data model so the hot path does not reconstruct a large object graph.

After a change, I measure the write cost as well. Indexes, materialized projections, denormalized fields and additional replicas all improve some reads by moving work elsewhere.

Concurrency and transaction boundaries

A transaction should protect one coherent consistency decision and then finish. I avoid remote network calls while database locks or snapshots are held. When a workflow crosses services, I prefer local transactions plus explicit messaging/outbox/saga patterns instead of attempting to stretch a database transaction across remote dependencies.

Concurrency failures are normal production behavior. Deadlocks, serialization failures, duplicate messages and failover retries need bounded retry policies and idempotent business behavior. Retrying blindly can duplicate a payment, booking, order or notification even if the database itself remains consistent.

High availability is application behavior

A replica, standby or cluster only provides infrastructure capability. The application still needs timeouts, reconnect behavior, read-consistency rules and a tested response to role changes.

I document which requests can tolerate stale data, which writes require stronger acknowledgement and what happens during a failover window. This is especially important for confirmation pages, inventory, payments and other workflows where users expect read-after-write behavior.

Backup and recovery

Replication is not a backup. A bad migration, accidental delete or corrupted logical state can replicate successfully.

For each production database I want a recovery-point objective and recovery-time objective. Backups are encrypted, retained independently and tested by restoring into another environment. The test is not complete when files are restored; it is complete when the application can connect and critical integrity checks pass.

Point-in-time recovery also needs enough log history—binary log, WAL or equivalent—to reach the target moment. Retention therefore needs to match the recovery policy.

Observability I expect

At database level I monitor query latency, throughput, active connections, connection-pool wait, replication lag, lock waits, storage growth and slow-query evidence. Engine-specific signals such as vacuum/bloat, buffer-pool behavior or document/index size are then layered on top.

At application level, database spans and metrics should identify the logical operation without emitting sensitive SQL parameters or document payloads. The goal is to connect a slow user request to a specific database operation and its saturation signal.

Failure modes I design against

• Promoting a standby without ensuring the old primary cannot accept writes. I treat this as a production risk because it can increase latency, widen the failure domain or make recovery behavior ambiguous.
• Using asynchronous replicas for immediate read-after-write workflows without accepting lag. I treat this as a production risk because it can increase latency, widen the failure domain or make recovery behavior ambiguous.
• Keeping WAL archives on the same failure domain as the primary. I treat this as a production risk because it can increase latency, widen the failure domain or make recovery behavior ambiguous.
• Measuring backup completion but never measuring restore completion. I treat this as a production risk because it can increase latency, widen the failure domain or make recovery behavior ambiguous.

These failure modes are useful review prompts because they turn a generic “database best practice” discussion into a concrete production scenario. If the team cannot explain how the system behaves under one of these conditions, that behavior is still an architectural unknown.

Deployment and migration strategy

Schema and index changes are production deployments. I prefer backward-compatible migrations that allow old and new application versions to overlap. Large index builds, backfills, partition changes or validation work are scheduled and monitored rather than hidden inside application startup.

When a change can create heavy I/O or locks, I test it against production-like volume and define a stop condition. A migration plan should include how to pause, roll forward or recover if runtime behavior differs from the estimate.

Production checklist

• Define synchronous versus asynchronous needs: the workload assumption and operational owner are documented.
• Monitor replay lag: the workload assumption and operational owner are documented.
• Fence the old primary during failover: the workload assumption and operational owner are documented.
• Preserve WAL for the recovery objective: the workload assumption and operational owner are documented.
• Use connection routing that understands roles: the workload assumption and operational owner are documented.
• Practice recovery to a new environment: the workload assumption and operational owner are documented.
• Query/operation p95 and p99 are observable.
• Connection pools have explicit maximums and wait metrics.
• Backup restore has been tested recently.
• Replica/standby lag has an alert threshold.
• Schema/index migrations have a rollback or roll-forward plan.
• Sensitive values are excluded from logs and telemetry.

Related guides

• Start with the PostgreSQL production architecture pillar
• PostgreSQL Production Architecture: Indexing, MVCC, Replication and Scaling
• PostgreSQL MVCC, Isolation, Locking and Deadlocks in Production

Closing perspective

For PostgreSQL, my rule is to choose structures from the workload outward: access pattern, consistency, concurrency, failure behavior, recovery and only then the specific database feature. That approach produces systems that remain understandable when data volume, traffic and team size grow.