On this page
Tracks

System Design — Distributed Logging

Last reviewed 14 Oct 2026

Part of the system design series. See the framework and building blocks first if you haven’t. Sibling problem: Metrics Monitoring & Alerting — metrics are numbers over time; logs are events with text. Indexing is covered in Search Engine.

1. Requirements

Functional

  • Collect logs from thousands of services/hosts/containers into one place.
  • Search by time range, service, level, request/trace id, and free text; tail logs live.
  • Alerts on log patterns (error spikes); dashboards from log-derived counts.
  • Retention: e.g. 7 days searchable fast, 30–90 days slower, 1+ year archived for compliance.

Non-functional

  • Ingest: ~1M log lines/s at peak (~1 KB each → ~1 GB/s, ~80 TB/day raw before compression).
  • Logging must never take down the app — if the pipeline is slow, apps drop or buffer logs; they don’t block.
  • Search over recent data in seconds; older data can take longer.
  • Cost is a first-class requirement — logging bills are often among the largest infrastructure costs.

2. High-level architecture

flowchart LR
APP[App containers<br/>write JSON to stdout] --> AG[Node agent<br/>Fluent Bit / Vector / OTel Collector]
AG -->|batched, compressed| K[(Kafka buffer)]
K --> PR[Processors<br/>parse, redact PII, enrich, sample]
PR --> HOT[(Hot store<br/>indexed, SSD, last 7 days)]
PR --> ARC[(Object storage<br/>compressed, long-term)]
HOT -->|age out| WARM[(Warm / cold tier)]
Q[Query UI / API] --> HOT
Q --> WARM
Q -.->|rehydrate on demand| ARC
PR --> AL[Alerting rules]
Agents ship to a buffer; indexers consume at their own pace; data ages through storage tiers
  • Apps log structured JSON to stdout; a per-node agent tails the files, adds metadata (service, pod, host, region) and ships in compressed batches. Apps never talk to the log backend directly.
  • Kafka buffer decouples producers from indexers. During an incident, log volume spikes 10× exactly when you need logs most — the buffer absorbs it instead of dropping data or crushing the indexers.
  • Processors parse, drop noise, redact PII/secrets, and sample high-volume debug logs.

3. Key design decision — what to index

Full-text indexing (Elasticsearch/OpenSearch, Splunk)Label-only indexing (Grafana Loki)Columnar (ClickHouse-style)
IndexEvery token in every lineOnly a few labels (service, env, level); content stored as compressed chunksColumns per field; sparse primary index, fast scans
QueryFast arbitrary text searchPick streams by labels, then brute-force grep the chunksFast filters and aggregations on structured fields
CostIndex often comparable in size to the data; expensive at scaleVery cheap storage; slower for needle-in-haystack across everythingCheap storage, excellent compression

The trade-off to state: full-text indexes make search fast but can roughly double storage and dominate ingest CPU. At high volume, many teams index only structured fields (service, level, trace_id, status) and scan the message text, or use label-only/columnar stores.

4. Deep dive — storage tiers and retention

  • Partition by time (daily or hourly indices/chunks) and by tenant/service. Time partitioning makes retention trivial: drop whole partitions instead of deleting rows.
  • Hot (recent days): SSD, replicated, fully indexed, fast queries.
  • Warm/cold: fewer replicas, cheaper disks, or searchable snapshots backed by object storage.
  • Archive: compressed files in S3/GCS (Parquet or compressed JSON), queryable on demand by rehydrating or with a query engine that reads object storage directly.
  • Compression is huge for logs (repetitive text): 10× or more is common with zstd and columnar layouts.

Volume control (logs grow faster than budgets):

  • Sample DEBUG/INFO in high-throughput paths; never sample ERROR.
  • Per-service ingest quotas, with rate-limit-and-count when exceeded (count dropped lines so you know).
  • Convert repetitive logs to metrics (“request served” lines → a counter).

5. What real systems do today

  • ELK/Elastic Stack (Elasticsearch + Logstash/Beats + Kibana) and its fork OpenSearch are the classic full-text approach, with index lifecycle management for hot/warm/cold/frozen tiers.
  • Grafana Loki indexes only labels and stores log chunks in object storage — designed explicitly to be cheaper than full-text indexing.
  • ClickHouse is widely used as a log backend for structured logs (columnar compression, fast aggregations).
  • OpenTelemetry standardizes collecting logs, metrics and traces, with the OTel Collector as agent/gateway; Fluent Bit and Vector are common lightweight agents.

6. Scaling & failure

  • Backend slow/down → agents buffer to local disk with a cap, then drop oldest with a counter; Kafka holds hours of data. Apps are never blocked.
  • Incident log storm → Kafka absorbs it; indexers lag but catch up; per-service quotas protect everyone else from one service in a crash loop.
  • Hot index shard overloaded (one noisy service) → route big tenants/services to their own indices or partitions.
  • Sensitive data leaked into logs → redaction in processors, restricted access per service, and the ability to purge a time range.
  • Clock skew → agents attach both the event’s timestamp and ingest time; query on event time.

Interview follow-ups

  • “Why put Kafka between agents and Elasticsearch?” — Absorbs incident spikes, decouples ingest from indexing, allows replay and multiple consumers (archive, alerting).
  • “Logs cost too much.” — Index fewer fields, sample, tier storage, compress, convert logs to metrics, enforce quotas.
  • “Find all logs for one request across services.” — Structured logs with trace_id propagated via OpenTelemetry context; exact-match field query.
  • “Full-text index vs Loki?” — Fast arbitrary search vs cheap storage and slower scans; choose by query patterns and budget.
  • “What if logging is down during an outage?” — Local buffering with caps, never blocking the app; Kafka retention for catch-up.

Further reading: Elastic documentation on index lifecycle management and data tiers · Grafana Loki documentation on its architecture and label design · OpenTelemetry documentation on logs and the Collector · ClickHouse blog posts on using ClickHouse for observability.