Heartbeat

A heartbeat is the simplest failure detector there is: every node periodically sends a small “I’m still here” message to a monitor or to its peers. If the heartbeats stop for long enough, the node is presumed dead and the system reacts: it elects a new leader, reroutes traffic or restarts something.

The tuning problem

Two numbers decide how a heartbeat behaves: the interval and the timeout (how many missed beats before you declare death).

• Too aggressive: a garbage-collection pause or a congested network makes a healthy node look dead. You fail over needlessly, and in the worst case end up with two nodes both believing they’re the leader.
• Too relaxed: a genuinely dead node keeps receiving traffic for ages.

There’s no perfect setting, because from the outside a slow node and a crashed node are indistinguishable. That’s a practical face of FLP Impossibility. More sophisticated detectors (such as the phi-accrual detector used in Cassandra and Akka) output a suspicion level based on the history of arrival times, rather than a binary alive/dead.

Where heartbeats show up

• Consensus: in Raft, the leader’s heartbeats stop followers starting an election. Silence triggers one.
• Orchestrators: container platforms run liveness probes (restart if it fails) and readiness probes (stop sending traffic until it passes). Mixing them up is a classic way to cause a restart loop.
• Long-lived connections: WebSockets use ping/pong frames, and Server-Sent Events (SSEs) streams send comment lines, both to detect dead peers and to stop proxies culling an idle connection.

A small example

A real-time UI pings its socket every 15 seconds and expects a pong within 5. Two missed pongs and it closes the socket, shows a subtle “reconnecting” indicator, and reconnects with backoff and jitter. Without the heartbeat, a half-open connection can sit there looking fine while delivering nothing, which on a price screen is the worst possible failure: stale data that looks live.

In practice

Every time I’ve debugged “the data just stopped updating”, the root cause was a connection that had died silently and nothing noticed. A heartbeat plus a visible connection state is cheap, and it turns a silent failure into a handled one. The wider taxonomy is in Types of Failure.