Key takeaways
- Logs, metrics and traces answer different questions, and a service needs all three from its first release.
- OpenTelemetry gives you vendor-neutral instrumentation, so changing monitoring tools does not mean instrumenting the code again.
- Service level objectives turn “is it working?” into a number engineering and the business have agreed.
- Alert on what users feel — errors and slowness — rather than on every resource metric, or people learn to ignore the pager.
- Every alert should link to a runbook that says what to check first.
Three signals, three questions
| Signal | Answers | From day one |
|---|---|---|
| Metrics | Is it healthy right now, and how is that trending? | Request rate, error rate and duration for every endpoint |
| Logs | What exactly happened in this case? | Structured entries, with a correlation ID on every line |
| Traces | Where did the time go across services? | Spans across each call between services and to the database |
The RED method — rate, errors, duration — is a good default for services, and the USE method — utilisation, saturation, errors — for the resources underneath them.
Instrument once, with an open standard
OpenTelemetry, a Cloud Native Computing Foundation project, provides vendor-neutral APIs, SDKs and a collector for metrics, logs and traces. Instrumenting with it means the monitoring backend can change later without touching the code — which matters in a long programme, or when a client mandates a particular tool.
Agree what “working” means
A service level objective states the reliability users should get — for example, 99.9% of checkout requests succeed within one second, measured over 28 days. The gap between that and perfection is the error budget. While budget remains, the team ships; when it is spent, reliability work comes first. That one rule settles arguments that would otherwise run on opinion.
Alerts people will not learn to ignore
- Page on symptoms users feel, such as errors and latency against the objective, not on CPU or disk alone.
- Alert on how fast the error budget is burning, so a slow leak and a sudden outage are both caught.
- Link every alert to a runbook that says what to check first and who to call.
- Review alerts every month, and delete the ones nobody acted on.
A first-sprint checklist
Correlation IDs everywhere
Generated at the edge, and passed through every call and every log line.
RED metrics per endpoint
Rate, errors and duration, on a dashboard the team actually opens.
Tracing through one collector
OpenTelemetry SDKs in each service, exported through a single collector.
One objective, one alert
For the journey that matters most, with its runbook linked.