Loki Fundamentals

How Loki Works

Loki is a log aggregation system designed for cost efficiency. Unlike Elasticsearch, Loki does not index the content of logs. It indexes only labels, making it much cheaper to operate. Log content is stored in compressed chunks on object storage.

Labels and Streams

In Loki, a stream is a unique combination of labels. Every log line belongs to exactly one stream. Labels define the stream identity, similar to Prometheus time series. Keep label cardinality low to avoid performance issues.

{app="api", namespace="production", pod="api-abc123"} line 1
{app="api", namespace="production", pod="api-def456"} line 2
{app="worker", namespace="production", pod="worker-789"} line 3

Chunks and Indexes

Loki stores log data in chunks. A chunk is a compressed block of log lines for a single stream. The index maps label combinations to chunks. Chunks are stored in object storage (S3, GCS, Azure Blob) while the index uses a compact store like BoltDB or a database.

Index: {app="api"} -> chunk_1, chunk_2, chunk_3
Chunk_1: [compressed log lines for time range T1-T2]
Chunk_2: [compressed log lines for time range T2-T3]
Chunk_3: [compressed log lines for time range T3-T4]

LogQL Selectors

LogQL selectors filter log streams by labels. They use the same syntax as Prometheus label matchers. Selectors are the first step in any LogQL query, narrowing the search to specific streams before processing log lines.

# All logs from the api app in production
{app="api", namespace="production"}

# Logs from api or web apps
{app=~"api|web"}

# Logs NOT from the default namespace
{namespace!="default"}

# All logs from pods matching a pattern
{pod=~"api-.*"}

Retention Configuration

Loki supports retention through compaction and deletion. Configure retention periods to control how long logs are stored. Older chunks are deleted automatically. Retention is critical for controlling storage costs.

limits_config:
  retention_period: 744h    # 31 days
  max_query_length: 721h    # Must be >= retention_period

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150