Observability at Scale

Monitoring and Alerting covered Uptime Kuma (is a service up or down) and a basic Grafana + Prometheus setup (metrics like CPU, RAM, disk over time). Both answer "what's the current state of things." Neither answers "what happened, in detail, across every service, at 3am last Tuesday" - that's what centralized logging is for, and it's the last new capability this guide covers.

Why centralized logs, once you have more than one machine

With one server, journalctl or docker compose logs on that box is enough - everything relevant is right there. Across a Proxmox cluster with several VMs each running their own containers, "check the logs" now means SSHing into several places and remembering which service lives where. Centralized logging pulls every container and system log into one searchable place, so debugging a cross-service issue (a failed deploy that cascades, an auth failure that shows up in three different services' logs) doesn't mean hopping between hosts.

Loki + Grafana

Loki (Grafana Labs) is a log aggregation system designed to pair with Grafana - it deliberately indexes only metadata (labels like container, host, service) rather than full-text indexing every log line, which keeps it far cheaper to run than something like Elasticsearch for the same volume of logs. You already have Grafana from the Intermediate monitoring setup, so Loki slots into the same dashboard rather than requiring a separate UI.

services:
  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-data:/loki
    restart: unless-stopped

  alloy:
    image: grafana/alloy:latest
    container_name: alloy
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./config.alloy:/etc/alloy/config.alloy
    command: run /etc/alloy/config.alloy
    ports:
      - "12345:12345"
    restart: unless-stopped

• Grafana Alloy is the agent that ships logs to Loki - one instance per host, reading local system and container logs and forwarding them with labels (which host, which container) attached. Alloy is the successor to Promtail, which Grafana Labs moved to long-term support in 2025 and ended entirely in March 2026 - don't stand up a new Promtail deployment at this point. Alloy also doubles as an OpenTelemetry Collector if you want that later, which Promtail never did.
• Add Loki as a data source in the Grafana you already have running, then logs and metrics dashboards live side by side.
• Pin image versions explicitly (3.0.0 for Loki above, not latest)
  • Loki's config schema has changed across major versions before, and an unplanned upgrade can break ingestion silently.

Long-term metrics: the storage trade-off

Prometheus (from the Intermediate setup) stores metrics at whatever resolution you scrape at - commonly every 15-30 seconds. That resolution is genuinely useful for debugging something that just happened, but it adds up fast: high-resolution data for a fleet of VMs and containers, kept indefinitely, can consume tens of gigabytes a year even for a modest homelab, and Prometheus's default local storage isn't built to be a long-term archive at that granularity.

The realistic options:

• Shorter local retention (Prometheus's default --storage.tsdb.retention.time, commonly 15 days) for high-resolution recent data, which covers "what happened this week" - the overwhelmingly common use case.
• Downsampling for anything kept longer - tools like Thanos or Mimir can store older data at reduced resolution (e.g. 5-minute averages instead of 15-second samples), shrinking storage dramatically for data you're unlikely to need at full precision months later.
• Just don't keep years of high-resolution data - for most homelabs, "is this trending up over months" is answerable from weekly or daily averages, not second-by-second history. Ask what question you're actually trying to answer with old metrics before provisioning storage to keep them all at full resolution.

Same judgment call applies to logs: Loki lets you set retention per label, so noisy, low-value logs (health-check pings, for instance) can expire in days while genuinely useful application logs are kept longer.

With logging, metrics, and their retention trade-offs sorted, the one scenario this guide hasn't touched yet is losing the entire site the homelab lives in - see Multi-Site Homelab for the honest version of that conversation.

Next: Multi-Site Homelab.