ES 索引与分片

摘要：分片规划是 Elasticsearch 最容易踩坑的一环——分片过多拖垮集群，分片过少导致数据倾斜和热点。本文给出分片容量估算方法、索引模板与 ILM 配置，以及热分片定位与 reindex/shrink 的实操步骤。

适用环境

curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cluster/health?pretty'
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/nodes?v&h=name,heap.percent,disk.used_percent'
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/indices?v' | head -10

操作步骤

1. 估算分片数

经验规则：单个分片大小控制在 20GB-50GB；分片总数不超过 数据节点数 × 每节点堆内存(GB) × 20（官方经验上限）。

# 现有索引的分片数与平均大小
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/indices?v&h=index,pri,rep,docs.count,store.size' | head -20

2. 创建索引并指定分片

curl -k -u elastic:'密码' -X PUT 'https://127.0.0.1:9200/logs-2026-10-09' -H 'Content-Type: application/json' -d '
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "30s"
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "level":      { "type": "keyword" },
      "message":    { "type": "text" },
      "trace_id":   { "type": "keyword" }
    }
  }
}'

3. 用索引模板统一规格（避免每个索引各自为政）

curl -k -u elastic:'密码' -X PUT 'https://127.0.0.1:9200/_index_template/logs-template' -H 'Content-Type: application/json' -d '
{
  "index_patterns": ["logs-*"],
  "priority": 100,
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "30s"
    },
    "mappings": {
      "properties": { "@timestamp": { "type": "date" }, "level": { "type": "keyword" } }
    }
  }
}'

4. 用 ILM 做生命周期管理（自动 rollover + 降冷 + 删除）

curl -k -u elastic:'密码' -X PUT 'https://127.0.0.1:9200/_ilm/policy/logs-policy' -H 'Content-Type: application/json' -d '
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "7d" } } },
      "warm":   { "min_age": "7d",  "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}'

5. 定位热点分片

# 各节点的 CPU / 磁盘 / 分片数是否均衡
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/allocation?v'
# 单节点分片数远超平均 = 分配倾斜
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/thread_pool/search?v&h=node_name,active,queue,rejected' 
# rejected > 0 说明该节点搜索线程池被打满，是典型热点信号

6. 用 routing 避免热点

# 写入时指定 routing，让同一业务键的数据固定落在同一分片
curl -k -u elastic:'密码' -X POST 'https://127.0.0.1:9200/logs-2026-10-09/_doc?routing=tenantA' \
  -H 'Content-Type: application/json' -d '{"level":"info","message":"hello"}'
# 查询时带上相同 routing，只命中一个分片，性能更好

7. 重建索引（调整分片数）

curl -k -u elastic:'密码' -X POST 'https://127.0.0.1:9200/_reindex?slices=auto&wait_for_completion=false' \
  -H 'Content-Type: application/json' -d '
{ "source": { "index": "logs-old" },
  "dest":   { "index": "logs-new", "op_type": "create" } }'

DANGER

主分片数创建后不可更改（number_of_shards 不可更新）。要改只能新建索引再 _reindex。副本数可以随时改，但副本数增加会瞬间触发大量数据复制，请在低峰期执行。

验证

# 1) 分片分布是否均衡
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/shards/logs-2026-10-09?v'

# 2) 无未分配分片
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/_cat/shards?v' | grep UNASSIGNED | wc -l

# 3) 写入并检索验证
curl -k -u elastic:'密码' -X POST 'https://127.0.0.1:9200/logs-2026-10-09/_doc' \
  -H 'Content-Type: application/json' -d '{"@timestamp":"2026-10-09T00:00:00Z","level":"info","message":"test"}'
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/logs-2026-10-09/_search?q=level:info&pretty'

# 4) ILM 状态
curl -k -u elastic:'密码' 'https://127.0.0.1:9200/logs-*/_ilm/explain?pretty' | head -20

常见坑

WARNING

按天建索引 + 每天 5 个分片，一年就是近 2000 个分片，会严重拖慢集群恢复速度。日志类场景请用 rollover 按大小滚动，而不是按天固定建索引。

WARNING

number_of_replicas=0 虽然省空间，但节点宕机即丢数据，且集群直接变 red。生产环境至少 1 副本。

DANGER

refresh_interval 设为 -1 会关闭自动刷新，写入性能最高但数据不可见；忘记改回来会导致"写入成功却搜不到"。只在大批量导入期间临时使用，导入后务必恢复。

参考资料

• Elasticsearch 分片与容量规划
• 索引模板官方文档
• ILM 生命周期管理
• Reindex API 文档