Fail closed: what auth should do when its own infrastructure breaks

Every distributed system eventually has the same conversation: a dependency is down, and something has to give. Most products choose availability, degrade gracefully, and write a postmortem. An authentication engine does not get that option, because the thing that gives is the security property itself.

The clearest example in ZKAuth is replay protection. Every login proof is single-use: once the engine accepts a proof, its hash goes into a registry, and any attempt to use it again is rejected as a replay attack. That registry is the last line of the verification pipeline, and it has to be atomic: two concurrent requests carrying the same proof must not both win.

INSERT INTO zkp_proof_registry (proof_hash, client_id, user_id, expires_at, created_at)VALUES ($1, $2, $3, $4, NOW())ON CONFLICT (proof_hash) DO NOTHING;-- rowCount === 0  ->  someone already used this proof. Reject.

The database enforces uniqueness, not application code. There is no check-then-set window, no lock to forget, no race to lose. If the insert affects zero rows, the proof was already spent and the login is refused.

The uncomfortable branch

The interesting engineering question is the failure branch: what if the registry itself is unreachable? Redis is down, or the database errors mid-write. The tempting answer is to log a warning and let the login through; users stay happy and the pager stays quiet. It is also exactly the window an attacker wants, because now replaying a captured proof works whenever they can induce (or wait for) an infrastructure blip.

In production, the ZKAuth engine refuses instead:

if (process.env.NODE_ENV === "production" && this.redisReplayProtection && !this.redisConnected) {  throw new ZKPError("Replay protection backend unavailable");}
// ...and the same rule for the database registry:if (process.env.NODE_ENV === "production") {  throw new ZKPError("Replay protection registry unavailable");}

Failing closed means a bad enough outage turns into rejected logins instead of silently weakened security. That is a real availability cost, chosen on purpose: an auth outage is recoverable and visible, while accepted replays are neither.

Tested like an attacker, not a demo

The unit suite for the proof service is written from the adversary’s side of the table. Among the cases that must pass on every commit: a replayed proof with a fresh nonce is still detected, only one of N concurrent identical proofs wins, stale and future-dated timestamps are rejected, tampered public signals fail on the tenant and email bindings, and the fail-closed branches actually throw when their backend is simulated away.

None of this is exotic. It is a short list of boring decisions, each made in the direction of integrity, enforced by the database and the test suite rather than by vigilance. That is what we mean when we say the engine fails closed.