深入解析与解决方案：处理Elasticsearch中all found copies are either stale or corrupt未分配分片问题

• 使用_cluster/allocation/explainAPI获取深入诊断信息：

curl -u 'es_user:es_user_passwd' -X GET "集群节点ip:9200/_cluster/allocation/explain?pretty" -H "Content-Type: application/json" -d'
{
  "index": "index_name",
  "shard": 分片ID,
  "primary": true
}'

• 响应中需要特别关注的字段：

• 步骤1：全面检查分片状态

curl -u 'es_user:es_user_passwd' -X GET "集群节点ip:9200/_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason,store" | grep -v STARTED

• 步骤2：识别最佳候选节点

• 步骤3：执行副本分配

curl -u 'es_user:es_user_passwd' -X POST "集群节点ip:9200/_cluster/reroute?retry_failed=true&pretty" -H "Content-Type: application/json" -d'
{
  "commands": [
    {
      "allocate_replica": {
        "index": "index_name",
        "shard": 0,
        "node": "集群节点"
      }
    }
  ]
}'

• 确认可以接受数据丢失：

curl -u 'es_user:es_user_passwd' -X GET "集群节点ip:9200/index_name/_stats?level=shards" | jq '.indices[].shards."0"'

• 执行stale主分片分配：

curl -u 'es_user:es_user_passwd' -X POST "集群节点ip:9200/_cluster/reroute?pretty" -H "Content-Type: application/json" -d'
{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "index_name",
        "shard": 未分配分片id,
        "node": "集群节点",
        "accept_data_loss": true
      }
    }
  ]
}'

• 数据同步过程

• 确认所有副本确实不可用：

curl -u 'es_user:es_user_passwd' -X GET "集群节点ip:9200/_cluster/allocation/explain?pretty" -H "Content-Type: application/json" -d'
{
  "index": "index_name",
  "shard": 未分配分片id,
  "primary": true
}' | grep -e "corrupt" -e "stale"

• 创建空主分片：

curl -u 'es_user:es_user_passwd' -X POST "集群节点ip:9200/_cluster/reroute?pretty" -H "Content-Type: application/json" -d'
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "index_name",
        "shard": 未分配分片id,
        "node": "集群节点",
        "accept_data_loss": true
      }
    }
  ]
}'

• 实时监控命令：

watch -n 1 'curl -u 'es_user:es_user_passwd' -s "集群节点ip:9200/_cat/shards/index_name?v&h=index,shard,prirep,state,node,docs|sort -k2,2n"'

• 文档数量比对：

# 获取历史文档数（如果有监控数据）
# 当前文档数
curl -u 'es_user:es_user_passwd' -s "集群节点ip:9200/index_name/_count?pretty" | jq '.count'

# 各分片文档数分布
curl -u 'es_user:es_user_passwd' -s "集群节点ip:9200/index_name/_stats/docs" | jq '.indices[].primaries.docs.count'

• 字段统计验证：

curl -u 'es_user:es_user_passwd' -X POST "集群节点ip:9200/index_name/_field_stats?fields=timestamp,user_id&level=cluster"

• 抽样数据检查：

curl -u 'es_user:es_user_passwd' -X GET "集群节点ip:9200/index_name/_search?size=100&q=*:*&sort=timestamp:desc"

• 索引设置建议：

{
  "index": {
    "number_of_replicas": 2,
    "unassigned.node_left.delayed_timeout": "10m",
    "recovery.retention_lease.period": "30m",
    "translog.durability": "async",
    "translog.sync_interval": "10s"
  }
}

• 分片分配感知配置：

• 配置命令：

curl -u 'es_user:es_user_passwd' -X PUT "集群节点ip:9200/_cluster/settings" -H "Content-Type: application/json" -d'
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "az",
    "cluster.routing.allocation.awareness.force.az.values": "zone1,zone2,zone3"
  }
}'

• 多级备份架构：

• 自动化快照策略：

# 创建快照策略
curl -u 'es_user:es_user_passwd' -X PUT "集群节点ip:9200/_slm/policy/daily-snapshots" -H "Content-Type: application/json" -d'
{
  "schedule": "0 30 2 * * ?", 
  "name": "<daily-snap-{now/d}>",
  "repository": "my_backup_repo",
  "config": {
    "indices": ["*"],
    "ignore_unavailable": true,
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}'

• Lucene索引检查工具：

# 在ES节点上执行
sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch-shard \
  -d /var/lib/elasticsearch/nodes/0/indices/your_index/0/index \
  -s /tmp/shard_recovery

• 修复流程：

• translog操作检查：

curl -u 'es_user:es_user_passwd' -X GET "集群节点ip:9200/your_index/_stats/translog?pretty"

• 使用_cat/recovery API：

curl -u 'es_user:es_user_passwd' -s "集群节点ip:9200/_cat/recovery/your_index?v&h=index,shard,time,type,stage,source_node,target_node,files_percent" | sort -k3,3