Today system observability is a much discussed term, which is often treated as “nice to have” during development, but becomes crucial once your system gets into production. It becomes even more important in case of distributed systems, where one can’t just spin up a local server and debug it end-to-end.

I’m going to share a experience of a development team, working in production microservice ecosystem, providing 24/7 support for the owned services, and how observability best-practices helped us to survive. We’ll touch the holy triad of observability: monitoring, log aggregation and distributed tracing, discuss some general best practices of how to make them work together and look at how we attempted to apply those best practices to real life.