$ cd ./topics/distributed-systems
Failure, consistency, consensus and delivery guarantees. The fundamentals underneath every platform and real-time system, written down so I actually understand them.
22 notes🌱 4 seed🌿 13 growing🌳 5 evergreen
Everything interesting eventually becomes a distributed system, including a browser talking to an API over a flaky connection. These notes are my working understanding of the fundamentals: what goes wrong, what “correct” even means when machines disagree, and how real systems agree anyway.
I’ve arranged them as a learning path, roughly in the order the ideas build on each other.
hover to explore · click to open
Case study — putting a declarative API gateway in front of a platform's microservices with little dedicated ops support. Versioned routes, stable names, one place for CORS, and validating locally before asking for anything.
A one-page, version-controlled record of a significant decision. Context, decision, consequences. Never edited after acceptance, only superseded, so the history of "why" survives.
When the network partitions, a distributed data store must choose between consistency and availability. PACELC adds the other half, that even without partitions you trade latency for consistency.
Eight assumptions everyone makes about networks, and every one of them is false. A checklist for any design that crosses a process boundary, including the browser.
At-most-once, at-least-once and "exactly-once". Why the network forces you to choose between losing messages and duplicating them, and how idempotency makes duplicates harmless.
A persistent, two-way channel between browser and server. Brilliant when both sides talk often; the price is stateful connections you now have to scale, heartbeat and reconnect.
A handstand is a feedback loop with a sensor, a delay and an actuator. So are autoscalers, rate limiters and a real-time UI under load — and they fail the same way.
Deadlock is everyone waiting on each other forever; starvation is one thread never getting a turn. Both are liveness failures, and the fixes are surprisingly mechanical.
In a fully asynchronous system, no deterministic algorithm can guarantee consensus if even one node may crash. Real systems get around it with timeouts and randomness, giving up guaranteed termination rather than safety.
A periodic "I'm alive" message that lets a system notice when a node has failed. Simple to build, subtle to tune, because a slow node and a dead one look the same.
"Don't communicate by sharing memory; share memory by communicating." Independent processes exchanging messages over channels, from Go and actors to queues and WebSockets.
Being able to ask new questions of a running system from its outputs — logs, metrics and traces, tied to the SLOs that turn them into decisions.
A consensus algorithm designed to be understandable. Elect a leader with randomised timeouts, replicate a log, and commit entries once a majority has them.
On a bad training day you drop to an easier progression rather than grinding ugly reps. Systems under stress should do the same — shed the expensive work, keep the core going.
Specific Adaptation to Imposed Demands — you get good at exactly what you train. Systems and test suites do too, so make the training look like production.
Specific Adaptation to Imposed Demands — you get good at exactly what you practise. Tests and staging only prepare a system for the conditions they simulate.
The SLI is what you measure, the SLO is your target for it, the SLA is the promise with consequences — and the error budget turns all three into release decisions.
Coordinating access to shared resources (critical sections, mutexes, semaphores, monitors) so concurrent code only ever does safe things.
Getting several nodes to agree on one value despite failures, defined by agreement, validity and termination. It's the problem underneath leader election, replicated logs and distributed locks, with Paxos as the classic answer.
Program vs process vs thread, and the three "multis" (multiprogramming, multiprocessing, multithreading): the units every concurrency discussion is built from.
A coordinator asks every participant to prepare, then tells them all to commit or abort. It gives atomic distributed transactions, but blocks if the coordinator fails at the wrong moment.
Crash, omission, timing and Byzantine failures, from "it stopped" to "it lies". Knowing which ones you're designing for decides how much machinery you need.