Observability: what to instrument first, and what to stop logging

The three signal types are not interchangeable and are routinely used for each other's jobs. Counting log lines to derive a rate, or logging every request in order to reconstruct a latency distribution, both work badly and cost more than doing it properly.

What each signal is actually for

The division is straightforward once stated. Metrics tell you something is wrong and alert on it. Traces tell you which component is responsible. Logs tell you what happened in the specific failing case. An incident normally moves through all three in that order, and a system missing any one of them makes that path longer.

The four signals worth having before anything else

If a service exposes only four measurements, these are the four. They cover the overwhelming majority of what goes wrong, and they are the ones an on-call engineer needs at 3am.

1. 01Latency, as a distribution rather than an average. Percentiles at 50, 95 and 99 — and keep failed requests separate, since a fast error flatters the figure.
2. 02Traffic, as requests per second by endpoint. Half of all incidents are explained by a change in the input rate.
3. 03Errors, as a rate rather than a count, split by class. Server faults and client rejections are different problems.
4. 04Saturation, meaning how full the most constrained resource is — connection pool, queue depth, memory, disk. This is the leading indicator the other three lack.

Saturation is the one most often absent, and the only one of the four that warns before users are affected. A connection pool at 95% utilisation has not caused an error yet and will within minutes under any additional load.

Structured logs, or logs you cannot query

A log line assembled by string interpolation can be read by a person and not queried by a machine. Since the entire reason for centralised logging is querying, the format is not a stylistic choice.

// Unqueryable: the fields exist only inside a sentence.
logger.info(`User ${userId} updated order ${orderId} in ${ms}ms`);

// Queryable: filter by tenant, aggregate durations, group by action.
logger.info({
  event: "order.updated",
  tenant_id: tenantId,
  user_id: userId,
  order_id: orderId,
  duration_ms: ms,
  trace_id: traceId,       // the field that ties logs to the trace
});

// Never in a log line, at any level:
//   passwords, tokens, keys, session identifiers, full card numbers,
//   or personal data beyond an identifier you can resolve internally.
// Logs are copied to more places than any other data in the system.

• One event name per kind of occurrence, stable over time, so a query written today still works next quarter
• A trace identifier on every line, which is what makes logs and traces one dataset instead of two
• Consistent field names across services — tenant_id everywhere, not tenantId in one service and tenant in another
• Log levels that mean something: warn for a situation needing attention eventually, error for one needing it now

Cardinality is why observability bills explode

Metrics are cheap because they aggregate. Each distinct combination of label values creates a separate series, so a label with unbounded values converts one cheap metric into millions of expensive ones.

The raw-URL row is the single most common cause of a surprise observability bill. Labelling by /orders/8fc1a2 rather than /orders/:id creates a new time series per order, permanently, and the cost grows with traffic rather than with the number of things being measured.

What to stop logging

• Successful requests at debug level in production. The metric already covers the aggregate; the line adds cost and no information
• Entry and exit of every function. This is what tracing does, properly and with sampling
• Full request and response bodies. Expensive, and the fastest route to logging personal data and credentials by accident
• Handled, expected conditions logged as errors. A validation rejection is not an error; logging it as one destroys the error rate as a signal
• Anything inside a hot loop. One badly placed line in a frequently executed path can outweigh the rest of the estate combined

The fourth item is worth the emphasis. Once the error log contains routine expected events, nobody can look at the error rate and tell whether something is broken — and the alert built on it gets muted, which is how a system with extensive logging ends up with no working alarm.

The order to build it in

1. 01The four signals per service, with latency as percentiles and requests kept separate from errors.
2. 02Structured logging with a trace identifier on every line and consistent field names.
3. 03Distributed tracing, sampled, once more than two services are involved in a request.
4. 04Alerts on the four signals, tied to user-visible symptoms rather than to resource utilisation.
5. 05A dashboard per service that answers "is this healthy" in one screen, with no interpretation needed.

Step four contains the discipline that makes the rest usable. An alert on high CPU wakes someone for a condition that may harm nobody; an alert on error rate or latency at the ninety-fifth percentile wakes them for something a user is experiencing. Alert on symptoms, investigate causes.