John Gallagher has a lovely 17-minute rant about observability. It says it is about Ruby on Rails, but it’s about all web apps ever.
- Wide events with a bunch of stuff, only 1-5 of them per HTTP request.
- Observability is a cultural change. Everyone on your team needs to know how to use the events and how to add to them.
- Convention over configuration — standardize your logs.
- Inch our way forward. Iterate.
- Record domain information, not just technical
- High cardinality: record IDs, not just status
- Structured logs, not freeform
- Implicit is ideal (automatic instrumentation)
- Use existing tools, and make them work as well as you can for you.
- Automatic is better than manual — automatically add stuff like your subscription ID.
- Don’t assume what your app is doing — be aware! with traces.
- Also be aware of how much it’s costing you.
- Exploration is better than dashboards. Get the specific answer you need, without distractions.
- Aim for predictability. “We can only do that by understanding it, and we can only understand it by observing it.”
