ELK Index Management

Managing Elasticsearch templates, ILM policies, rollover aliases, shards, and Kibana data views.

8 min readObservability & Monitoring StackOperate

ELK Architecture

The ELK stack (Elasticsearch, Logstash, Kibana) is a full-text search and analytics platform. Unlike Loki, Elasticsearch indexes the content of logs, enabling powerful full-text search. The tradeoff is higher storage and operational cost.

Component	Role	Port
Elasticsearch	Storage and search	9200
Logstash/Beats	Ingestion and processing	5044
Kibana	Visualization	5601

Index Templates

Index templates define mappings and settings for indices that match a pattern. When a new index is created matching the pattern, Elasticsearch applies the template automatically. This ensures consistent field types and settings.

Index template definition


{
  "index_patterns": ["app-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "app-logs-policy"
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "message": { "type": "text" },
        "trace_id": { "type": "keyword" }
      }
    }
  }
}

ILM Policies

Index Lifecycle Management (ILM) policies automate index lifecycle stages: hot (active writes), warm (read-only), cold (compressed), and delete. ILM reduces storage costs by automatically moving old data to cheaper storage tiers.

ILM policy


{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Rollover Aliases

Rollover aliases write to the latest index while maintaining a consistent name for reads. When an index reaches its size or age limit, Elasticsearch creates a new index and updates the alias. This is essential for ILM.

Rollover alias setup


# Create initial index with alias
PUT app-logs-000001
{
  "aliases": {
    "app-logs": { "is_write_index": true },
    "app-logs-read": {}
  }
}

# Rollover creates new index and updates alias
POST app-logs/_rollover
{
  "conditions": {
    "max_primary_shard_size": "30gb",
    "max_age": "1d"
  }
}

Shard Management

Shards are the basic unit of scalability in Elasticsearch. Each shard is a Lucene index. Too many shards waste memory. Too few shards limit parallelism. The general rule is to keep shard size between 10-50 GB.

Do not over-shard

Each shard uses memory and file descriptors. A cluster with 10,000 small shards performs worse than one with 1,000 properly sized shards. Right-size your shards based on data volume and retention period.

Kibana Data Views

Data views (formerly index patterns) tell Kibana which indices to search. Create data views matching your index patterns. Use field formatters to customize how fields appear in Kibana.

Create data view via API


POST kibana/api/data_views/data_view
{
  "data_view": {
    "title": "app-logs-*",
    "name": "Application Logs",
    "timeFieldName": "timestamp"
  }
}

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

Logging Operations