Skip to content
matthewcm.dev

    ↑ ↓ move · ↵ open · tab scopeopen graph →

    Observability

    Being able to ask new questions of a running system from its outputs — logs, metrics and traces, tied to the SLOs that turn them into decisions.

    Growingtended planted 2 min read
    On this page

    Monitoring tells you whether the things you predicted would break have broken. Observability is the ability to ask questions you didn’t predict — “why are checkout requests from one region slow only for logged-in users?” — using only what the system emits. In a distributed system, where the Fallacies of Distributed Computing guarantee surprises, that difference matters.

    The three signals

    SignalWhat it isGood for
    LogsTimestamped events, ideally structured (JSON, not prose)The detail of what happened in one place
    MetricsNumbers aggregated over time: rates, counts, percentilesTrends, dashboards, cheap alerting
    TracesOne request’s path across services, as a tree of timed spansWhere the time or the error went in a multi-service call

    They’re far more useful joined up: a trace ID in every log line lets you jump from a spike on a dashboard to the exact requests behind it.

    The pipeline

    1. Instrument — OpenTelemetry has become the vendor-neutral standard for emitting all three signals.
    2. Collect and ship — agents and collectors (Fluentd, the OTel Collector) buffer and forward telemetry.
    3. Store and query — a metrics store, a log index, a trace backend.
    4. Alert on what users feel — not on CPU at 80%, but on the indicators behind your SLOs.

    The frontend counts too

    Observability doesn’t stop at the API. For UIs I want error tracking with source maps (Sentry-style), real-user performance metrics (Core Web Vitals), and for real-time interfaces, client-side measures like dropped frames and message lag. A trading screen can have perfect backend latency and still feel broken because the render loop can’t keep up — see Zustand Transient Updates.

    Trade-offs

    • Cost and noise. Logging everything is expensive and makes the signal harder to find. Sample traces; keep high-cardinality detail in traces and logs, not metrics.
    • Alert fatigue. Alerts that don’t require a human action train people to ignore alerts. Page on symptoms, not causes.
    • Privacy. Telemetry is data. Scrub tokens and personal data at the source.

    In practice

    The habit I’ve built is to treat observability as part of the definition of done: a feature isn’t finished until I can tell, from the outside, whether it’s working for users. The same idea applies to delivery itself — lead time and change failure rate are the telemetry of your process. And it applies to agents: if you can’t see what an AI agent perceived and why it acted, you can’t trust it.

    • #observability
    • #monitoring
    • #logs
    • #metrics
    • #tracing
    • #opentelemetry
    • DORA MetricsGrowing
      DORA metrics are Observability for the delivery process, and the scorecard for an Internal Developer Platform: if the platform is working, these numbers move.
    • Fallacies of Distributed ComputingEvergreen
      …etries and a "simple" polling design that would have hammered a backend. When something does go wrong anyway, Observability is how you find out which fallacy it was.
    • Internal Developer PlatformGrowing
      Service scaffolding (Golden Paths) CI/CD and environments Infrastructure provisioning, often through infrastructure-as-code modules Secrets management and access control Observability wired in by default (Observability)
    • Mining Reddit for ResearchGrowing
      …erything, stable IDs, and a clear line between raw and cleaned data. The research equivalent of good logging (Observability).
    • Regress, Don't Grind: Graceful DegradationSeed
      In training, you notice the bad day. Systems need instruments to notice for them: latency percentiles, error rates, queue depth, dropped frames on the client. Without that Observability, you're grinding blind.
    • Rolling Out an API GatewayGrowing
      …of per-route latency and error rates on day one. The gateway sees every request, which makes it the cheapest Observability point in the whole system, and I added that view later than I should have.
    • SLA vs SLO vs SLIGrowing
      …ould tell us this is working for users?" — and the answer became an SLI. The plumbing behind those numbers is Observability.
    • What Makes an AI AgentGrowing
      …fects hurt agents even more than they hurt people, because agents won't ask in Slack when the docs are wrong. Observability isn't optional. You need to see what the agent perceived and why it chose an action — the same t…

    ↪ Links to (4)