The one-line answer and the two questions inside it

Ask how a database survives a primary failure and you will usually get the one-line answer: "if the leader dies, we promote a follower." It is true, and it hides the problem in two words. Dies: how do you know it died? We: who has authority to decide?

Over an asynchronous network, a crashed leader and a slow leader can look identical to an observer. Silence can reflect a crash, scheduler or I/O delay, or a partition. FLP formalizes a related limit: in a fully asynchronous model, deterministic consensus cannot be guaranteed to terminate with even one faulty process because a crashed process cannot be distinguished from a sufficiently slow one. FLP does not by itself describe an operational failover system, but it explains why a timeout cannot prove a crash. Safe failover therefore needs separate rules for promotion authority and for preventing a superseded primary from accepting protected writes.

To keep this concrete, carry one setup through every failure mode: a PostgreSQL primary A, one asynchronous streaming replica B, and a promoter that polls A every two seconds and calls pg_promote() on B after three consecutive failed observations.

Detection is a guess with a deadline

The third failed observation triggers promotion. At two-second intervals, the first and third failed observations are four seconds apart; total detection delay also depends on where the failure falls in the poll cycle and how long each check takes. The primary might be down, delayed by scheduling, I/O, or a checkpoint, or partitioned from the promoter while still reachable by application clients.

Lease-based designs make the deadline explicit. The leader holds, for example, a 10-second lease and may act only while it remains valid under a suitable clock and deadline model. Correct use requires validation immediately around each protected operation, without an unbounded check-to-use gap. A holder can validate its lease and then pause before or during I/O; revalidation after resumption cannot retract an operation the resource already accepted. Lease checks therefore need resource-side fencing or completed isolation when in-flight work can outlive the lease.

Authority: who is allowed to promote

If one promoter acts on its suspicion alone, it becomes another single point of failure. It can also be partitioned from A while clients are not. Promoting from that observer's timeout can create two primaries.

An integrated consensus protocol provides stronger election safety only with its required rules. Quorum intersection and durable one-vote-per-term records prevent two leaders in one term. Monotonically increasing terms and candidate-log freshness constrain later leaders to eligible histories. Membership must remain stable or change through a joint-consensus procedure. External promoters that merely count failed health checks do not inherit those guarantees unless their coordination layer provides comparable epoch, voting, and membership rules. Even then, election chooses a leader; it does not disable writes to the old PostgreSQL primary.

Split brain: two nodes, both locally correct

Suppose the promoter is partitioned from A and promotes B. A remains alive and serves clients that can reach it. Both PostgreSQL nodes accept writes into their own local WAL timelines.

sequenceDiagram
    participant C1 as Clients (side A)
    participant A as Old primary A
    participant B as Promoted replica B
    participant C2 as Clients (side B)
    Note over A,B: Partition: A and B cannot reach each other
    C1->>A: UPDATE x = 1
    A-->>C1: commit acknowledged
    C2->>B: UPDATE x = 2
    B-->>C2: commit acknowledged
    Note over A,B: Two acknowledged local histories of x.<br/>PostgreSQL does not merge the timelines.

Both clients were told that their transaction committed on the node they reached. Asynchronous replication does not place both writes in a common history, and PostgreSQL cannot automatically merge divergent timelines. Recovery requires an operator to determine which acknowledged old-timeline changes must be preserved and how to reconcile them with the new primary.

The zombie leader and divergent timelines

The old primary may have been unreachable rather than dead and can continue accepting traffic through stale client or proxy routes. In this shared-nothing PostgreSQL topology, A cannot directly overwrite B's local storage. The failure boundary is the set of clients still writing to A and receiving acknowledgments for changes that remain on its divergent timeline.

sequenceDiagram
    participant C as Stale client or proxy route
    participant A as Old primary A
    participant P as Promoter
    participant B as New primary B
    participant O as Operator
    Note over A,B: Partition isolates A from promoter and B
    P->>A: Health checks fail (3x)
    P->>B: pg_promote()
    C->>A: UPDATE balance = 500
    A-->>C: commit acknowledged on old timeline
    B->>B: UPDATE balance = 900
    Note over A,B: WAL timelines have diverged
    Note over A,B: Partition heals
    O->>A: Stop writes and preserve needed old-timeline changes
    O->>A: pg_rewind if eligible, or rebuild
    A->>B: Follow B as the primary

The write on A remains there; it does not silently land on B. When connectivity returns, routes to A must be stopped before recovery. If its acknowledged writes matter, they need deliberate preservation and application-level reconciliation onto the chosen history. The old primary can then use pg_rewind where its prerequisites are met and the required WAL is available, or be rebuilt from a fresh base backup. pg_rewind realigns the data files with B's timeline; it does not merge the conflicting logical changes.

Why ordinary tests can miss this

These failures require a partition, promotion, stale route, and particular ordering of writes. A test that only kills the primary process exercises a different boundary. Even a fault-injection suite may miss the case if it does not keep the old primary writable while the new one accepts traffic. Testing should exercise that interleaving and verify the intended policy for acknowledged writes.

Jepsen's Elasticsearch analysis documented acknowledged writes lost during network partitions. In GitHub's October 2018 incident, a 43-second network partition triggered automated MySQL failover activity to the US West Coast. East Coast primaries held writes that had not replicated west, including 954 on one busy cluster, while the new West Coast primaries accepted other writes. GitHub could not simply fail back without addressing the divergence, and service remained degraded for 24 hours and 11 minutes.

Three controls, three different questions

The practical safeguards address different boundaries and are commonly combined.

Quorum-based election answers "who leads." Under the term, voting, log, and membership conditions above, consensus can select an eligible leader without electing two leaders in one term. That still does not constrain a deposed PostgreSQL primary's write path.

Fencing tokens answer "may this write land." A complete fencing design uses a linearizable issuer for monotonically increasing epochs. Before the new leader writes, its epoch must be durably installed at every protected target. Each target persistently stores the highest epoch it has accepted and atomically compares the supplied epoch with that value as part of each mutation, rejecting stale operations. Kleppmann's analysis of distributed locking explains this contract; antirez's response argues that a resource capable of that check may support a simpler unique-token compare-and-set design.

Ordinary PostgreSQL writes do not natively carry and enforce arbitrary client fencing epochs. The carried topology therefore needs completed infrastructure fencing before promotion, or a mandatory proxy or resource that implements the epoch check and cannot be bypassed. During a partition, A need not learn that it was superseded if it has been isolated from protected resources or its old epoch is rejected.

STONITH answers "can the old node still write." Shoot The Other Node In The Head powers off or isolates A through a channel it cannot veto. Isolation must complete and be verified before B is promoted. This supports resources that cannot compare epochs, but it can unnecessarily remove a healthy node during a communication fault and adds a dependency on the fencing system.

One decision sits underneath these controls. Fencing prevents further old-primary writes, but it does not preserve acknowledged changes that never replicated. Synchronous replication configured to wait for B to durably receive relevant WAL can preserve those acknowledgments across promotion to B, at the cost of additional write latency and reduced write availability when B is unavailable. Asynchronous replication avoids that dependency while accepting possible loss or reconciliation of the unreplicated tail. The appropriate policy depends on which outcome the application can absorb.

The sentence worth saying in an interview

For this PostgreSQL setup, a concise answer is: "I would not promote the replica until the old primary is fenced. If every protected resource supports fencing epochs, each write must carry the current epoch and stale epochs must be rejected atomically." That statement should lead into architecture-specific questions: whether the election has durable terms and votes, whether the database or a mandatory proxy can enforce epochs, how routing is withdrawn, and what an acknowledgment promises under the chosen replication mode.

---

Formation publishes this blog. If you want practice defending decisions at these failure boundaries, Formation's Senior+ track offers feedback from experienced engineers who can challenge the assumptions in your design.