Types of Failure

“Design for failure” is good advice that doesn’t say which failures. Distributed systems theory sorts them into a ladder, from easiest to handle to hardest. Every rung up needs more machinery.

The ladder

Failure	What happens	Example
Crash-stop	A node stops and never comes back	A container killed and not restarted
Crash-recovery	A node stops, then returns, possibly having lost in-memory state	A server reboots mid-request
Omission	Some messages are lost: sent but never received, or never sent	A dropped packet; a full queue silently discarding
Timing	Responses arrive, but too late to be useful	A GC pause makes a healthy node look dead
Byzantine	A node behaves arbitrarily: corrupts data, contradicts itself, lies	Faulty hardware, a bug, or a malicious participant

Why the distinction matters

• Crash failures can be handled by replication and a way to notice the crash, usually a Heartbeat.
• Omission needs retries, which means duplicates, which means idempotency (Message Delivery Semantics).
• Timing failures are the sneaky ones, because a slow node and a dead node look the same from outside. That ambiguity is the heart of FLP Impossibility.
• Byzantine tolerance is expensive: classic protocols need more than three times as many nodes as the failures they tolerate. Most internal systems assume nodes are honest and only crash. Blockchains are the famous exception, because there participants really might lie.

Partial failure is the default

On one machine, things mostly work or crash entirely. In a distributed system, part of the system fails while the rest carries on, and often nobody can tell which part. Designing well means being explicit: “we tolerate crash-recovery and omission, not Byzantine” is a design decision worth writing in an ADR (Architecture Decision Records).

In practice

The frontend has its own version of this ladder. A request can fail fast (crash), never return (omission), return after the user has given up (timing), or return garbage from a misbehaving proxy (a small Byzantine taste). Timeouts, retries with limits, schema validation at the boundary (API Modelling) and clear loading and error states are the same ideas at a smaller scale. Designing what happens after a failure is Regress, Don’t Grind: Graceful Degradation.