https://heezy.blog/tags/loki/Recent content in Loki on heezy.blogHugoen-usSun, 22 Mar 2026 00:00:00 +0000https://heezy.blog/posts/lgtm-monitoring-stack/Sun, 22 Mar 2026 00:00:00 +0000https://heezy.blog/posts/lgtm-monitoring-stack/<p>The monitoring stack runs on a dedicated VM at 10.x.x.x (shared-lgtm), deliberately separate from the Kubernetes cluster it watches. If the cluster goes down, the thing watching it needs to still be running. Seven containers in a single Docker Compose stack handle metrics collection, log aggregation, long-term storage, tracing, and dashboards with alerting. The whole thing is deployed and configured by a single Ansible role (<code>roles/lgtm/</code>) with Jinja2 templates for every config file. Push a change, GitHub Actions runs the playbook, Ansible templates the configs and restarts the stack. This post covers what each component does, what gets scraped, and how alerting works.</p>