第二章 ELK安装部署与环境配置-EW帮帮网

环境准备

系统要求

硬件要求：

最小配置:
  CPU: 2核心
  内存: 4GB
  磁盘: 20GB
  网络: 1Gbps

推荐配置:
  CPU: 8核心
  内存: 32GB
  磁盘: 500GB SSD
  网络: 10Gbps

生产环境:
  CPU: 16核心+
  内存: 64GB+
  磁盘: 1TB+ NVMe SSD
  网络: 10Gbps+

操作系统支持：

Linux (推荐): CentOS 7+, Ubuntu 18.04+, RHEL 7+
Windows: Windows Server 2016+
macOS: 10.14+

Java环境配置

安装OpenJDK 11/17：

# CentOS/RHEL
sudo yum install java-11-openjdk java-11-openjdk-devel

# Ubuntu/Debian
sudo apt update
sudo apt install openjdk-11-jdk

# 验证安装
java -version
javac -version

环境变量配置：

# 编辑环境变量
sudo vim /etc/environment

# 添加以下内容
JAVA_HOME=/usr/lib/jvm/java-11-openjdk
PATH=$PATH:$JAVA_HOME/bin

# 重新加载环境变量
source /etc/environment

# 验证配置
echo $JAVA_HOME

系统优化

内核参数调优：

# 编辑系统限制
sudo vim /etc/security/limits.conf

# 添加以下内容
elastic soft nofile 65536
elastic hard nofile 65536
elastic soft nproc 4096
elastic hard nproc 4096
elastic soft memlock unlimited
elastic hard memlock unlimited

# 编辑系统参数
sudo vim /etc/sysctl.conf

# 添加以下内容
vm.max_map_count=262144
vm.swappiness=1
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=65535

# 应用配置
sudo sysctl -p

创建用户和目录：

# 创建elastic用户
sudo useradd -m -s /bin/bash elastic

# 创建安装目录
sudo mkdir -p /opt/elastic
sudo chown -R elastic:elastic /opt/elastic

# 创建数据目录
sudo mkdir -p /var/lib/elasticsearch
sudo mkdir -p /var/log/elasticsearch
sudo chown -R elastic:elastic /var/lib/elasticsearch
sudo chown -R elastic:elastic /var/log/elasticsearch

Elasticsearch安装

1. 下载和安装

使用包管理器安装：

# 添加Elastic仓库
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# 更新包列表
sudo apt update

# 安装Elasticsearch
sudo apt install elasticsearch

手动下载安装：

# 下载Elasticsearch
cd /opt/elastic
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.0-linux-x86_64.tar.gz

# 解压
tar -xzf elasticsearch-8.11.0-linux-x86_64.tar.gz
mv elasticsearch-8.11.0 elasticsearch

# 设置权限
chown -R elastic:elastic elasticsearch

2. 配置文件

主配置文件 (elasticsearch.yml)：

# ======================== Elasticsearch Configuration =========================

# ---------------------------------- Cluster -----------------------------------
cluster.name: elk-cluster

# ------------------------------------ Node ------------------------------------
node.name: node-1
node.roles: [ master, data, ingest ]

# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# ---------------------------------- Memory ------------------------------------
bootstrap.memory_lock: true

# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

# --------------------------------- Discovery ----------------------------------
discovery.type: single-node
# discovery.seed_hosts: ["host1", "host2"]
# cluster.initial_master_nodes: ["node-1", "node-2"]

# ---------------------------------- Security ----------------------------------
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

# ---------------------------------- Various -----------------------------------
action.destructive_requires_name: true
indices.query.bool.max_clause_count: 10000

JVM配置 (jvm.options)：

# JVM heap size
-Xms4g
-Xmx4g

# GC configuration
-XX:+UseG1GC
-XX:G1HeapRegionSize=16m
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:+UseZGC

# Memory settings
-XX:+AlwaysPreTouch
-Xss1m

# GC logging
-Xlog:gc*,gc+age=trace,safepoint:gc.log:utctime,pid,tid,level
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=32
-XX:GCLogFileSize=64m

# Heap dumps
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch

# Security
-Djava.security.policy=all.policy

3. 启动和验证

启动服务：

# 使用systemd启动
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
sudo systemctl status elasticsearch

# 手动启动
su - elastic
cd /opt/elastic/elasticsearch
./bin/elasticsearch -d

验证安装：

# 检查集群状态
curl -X GET "localhost:9200/_cluster/health?pretty"

# 检查节点信息
curl -X GET "localhost:9200/_nodes?pretty"

# 检查索引
curl -X GET "localhost:9200/_cat/indices?v"

Kibana安装

1. 下载和安装

# 使用包管理器
sudo apt install kibana

# 手动安装
cd /opt/elastic
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.11.0-linux-x86_64.tar.gz
tar -xzf kibana-8.11.0-linux-x86_64.tar.gz
mv kibana-8.11.0 kibana
chown -R elastic:elastic kibana

2. 配置文件

主配置文件 (kibana.yml)：

# =================== System: Kibana Server ===================
server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana-server"

# =================== System: Elasticsearch ===================
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password"

# =================== System: SSL ===================
server.ssl.enabled: true
server.ssl.certificate: "/path/to/kibana-server.crt"
server.ssl.key: "/path/to/kibana-server.key"

elasticsearch.ssl.certificateAuthorities: ["/path/to/ca.crt"]
elasticsearch.ssl.verificationMode: certificate

# =================== System: Logging ===================
logging.appenders:
  file:
    type: file
    fileName: /var/log/kibana/kibana.log
    layout:
      type: json

logging.root:
  appenders:
    - default
    - file
  level: info

# =================== System: Other ===================
pid.file: /var/run/kibana/kibana.pid
path.data: /var/lib/kibana

# =================== System: Security ===================
xpack.security.encryptionKey: "something_at_least_32_characters"
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"
xpack.reporting.encryptionKey: "something_at_least_32_characters"

# =================== System: Monitoring ===================
monitoring.ui.container.elasticsearch.enabled: true
monitoring.ui.container.logstash.enabled: true

3. 启动和验证

# 启动Kibana
sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl status kibana

# 检查日志
sudo tail -f /var/log/kibana/kibana.log

# 访问Web界面
# http://localhost:5601

Logstash安装

1. 下载和安装

# 使用包管理器
sudo apt install logstash

# 手动安装
cd /opt/elastic
wget https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz
tar -xzf logstash-8.11.0-linux-x86_64.tar.gz
mv logstash-8.11.0 logstash
chown -R elastic:elastic logstash

2. 配置文件

主配置文件 (logstash.yml)：

# =================== Node identity ===================
node.name: logstash-1

# =================== Data path ===================
path.data: /var/lib/logstash
path.logs: /var/log/logstash
path.settings: /etc/logstash

# =================== Pipeline Settings ===================
pipeline.workers: 4
pipeline.batch.size: 1000
pipeline.batch.delay: 50

# =================== Pipeline Configuration ===================
path.config: /etc/logstash/conf.d/*.conf
config.reload.automatic: true
config.reload.interval: 3s

# =================== Logging ===================
log.level: info
path.logs: /var/log/logstash

# =================== HTTP API ===================
http.host: "0.0.0.0"
http.port: 9600

# =================== Monitoring ===================
monitoring.enabled: true
monitoring.elasticsearch.hosts: ["https://localhost:9200"]
monitoring.elasticsearch.username: logstash_system
monitoring.elasticsearch.password: your_password

管道配置示例 (/etc/logstash/conf.d/apache.conf)：

input {
  beats {
    port => 5044
  }
  
  file {
    path => "/var/log/apache2/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  if [fields][log_type] == "apache" {
    grok {
      match => { 
        "message" => "%{COMBINEDAPACHELOG}" 
      }
    }
    
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    
    geoip {
      source => "clientip"
      target => "geoip"
    }
    
    useragent {
      source => "agent"
      target => "useragent"
    }
    
    mutate {
      convert => { "response" => "integer" }
      convert => { "bytes" => "integer" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "apache-logs-%{+YYYY.MM.dd}"
    user => "logstash_writer"
    password => "your_password"
    ssl => true
    cacert => "/path/to/ca.crt"
  }
  
  stdout {
    codec => rubydebug
  }
}

3. JVM配置

JVM设置 (jvm.options)：

# Heap size
-Xms2g
-Xmx2g

# GC settings
-XX:+UseG1GC
-XX:+UseStringDeduplication

# Memory settings
-XX:+AlwaysPreTouch

# GC logging
-Xlog:gc*,gc+age=trace,safepoint:gc.log:utctime,pid,tid,level

# Heap dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/logstash

Beats安装

1. Filebeat安装

# 下载和安装
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-linux-x86_64.tar.gz
tar -xzf filebeat-8.11.0-linux-x86_64.tar.gz
mv filebeat-8.11.0-linux-x86_64 /opt/elastic/filebeat

Filebeat配置 (filebeat.yml)：

# =================== Filebeat inputs ===================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/*.log
    - /var/log/nginx/*.log
  fields:
    log_type: web
  fields_under_root: true
  multiline.pattern: '^\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
    - /var/log/app/*.log
  fields:
    log_type: application
  fields_under_root: true

# =================== Filebeat modules ===================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

# =================== Processors ===================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# =================== Outputs ===================
output.logstash:
  hosts: ["localhost:5044"]
  
# output.elasticsearch:
#   hosts: ["https://localhost:9200"]
#   username: "filebeat_writer"
#   password: "your_password"
#   ssl.certificate_authorities: ["/path/to/ca.crt"]

# =================== Logging ===================
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

2. Metricbeat安装

# 下载和安装
wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.11.0-linux-x86_64.tar.gz
tar -xzf metricbeat-8.11.0-linux-x86_64.tar.gz
mv metricbeat-8.11.0-linux-x86_64 /opt/elastic/metricbeat

Metricbeat配置 (metricbeat.yml)：

# =================== Metricbeat modules ===================
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

# =================== System module ===================
metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - socket_summary
    - filesystem
    - fsstat
  enabled: true
  period: 10s
  processes: ['.*']
  
- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

# =================== Outputs ===================
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "metricbeat_writer"
  password: "your_password"
  ssl.certificate_authorities: ["/path/to/ca.crt"]

# =================== Processors ===================
processors:
  - add_host_metadata: ~
  - add_docker_metadata: ~

集群配置

1. Elasticsearch集群

Master节点配置：

# elasticsearch.yml for master node
cluster.name: elk-production
node.name: master-1
node.roles: [ master ]

network.host: 192.168.1.10
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["192.168.1.10", "192.168.1.11", "192.168.1.12"]
cluster.initial_master_nodes: ["master-1", "master-2", "master-3"]

gateway.expected_master_nodes: 3
gateway.expected_data_nodes: 6
gateway.recover_after_master_nodes: 2
gateway.recover_after_data_nodes: 4

Data节点配置：

# elasticsearch.yml for data node
cluster.name: elk-production
node.name: data-1
node.roles: [ data, ingest ]

network.host: 192.168.1.20
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["192.168.1.10", "192.168.1.11", "192.168.1.12"]

# 数据节点特定配置
indices.memory.index_buffer_size: 30%
indices.memory.min_index_buffer_size: 96mb
indices.fielddata.cache.size: 40%

2. 负载均衡配置

Nginx配置：

upstream elasticsearch {
    server 192.168.1.20:9200 max_fails=3 fail_timeout=30s;
    server 192.168.1.21:9200 max_fails=3 fail_timeout=30s;
    server 192.168.1.22:9200 max_fails=3 fail_timeout=30s;
}

upstream kibana {
    server 192.168.1.30:5601 max_fails=3 fail_timeout=30s;
    server 192.168.1.31:5601 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name elasticsearch.example.com;
    
    location / {
        proxy_pass http://elasticsearch;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 30s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;
    }
}

server {
    listen 80;
    server_name kibana.example.com;
    
    location / {
        proxy_pass http://kibana;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 30s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;
    }
}

安全配置

1. 启用安全功能

# 生成证书
cd /opt/elastic/elasticsearch
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

# 设置密码
./bin/elasticsearch-setup-passwords interactive

2. 用户和角色管理

# 创建自定义角色
curl -X POST "localhost:9200/_security/role/logstash_writer" -H 'Content-Type: application/json' -d'
{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"],
  "indices": [
    {
      "names": [ "logstash-*" ],
      "privileges": ["write","create","create_index","manage","manage_ilm"]
    }
  ]
}'

# 创建用户
curl -X POST "localhost:9200/_security/user/logstash_internal" -H 'Content-Type: application/json' -d'
{
  "password" : "your_password",
  "roles" : [ "logstash_writer" ],
  "full_name" : "Internal Logstash User"
}'

性能调优

1. 系统级优化

# 禁用swap
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# 文件描述符限制
echo "elastic soft nofile 65536" >> /etc/security/limits.conf
echo "elastic hard nofile 65536" >> /etc/security/limits.conf

# 虚拟内存设置
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p

2. Elasticsearch优化

# elasticsearch.yml优化配置
bootstrap.memory_lock: true
indices.memory.index_buffer_size: 30%
indices.memory.min_index_buffer_size: 96mb
indices.fielddata.cache.size: 40%
indices.queries.cache.size: 10%
indices.requests.cache.size: 2%

# 线程池配置
thread_pool:
  write:
    size: 8
    queue_size: 1000
  search:
    size: 13
    queue_size: 1000

监控配置

1. 集群监控

# 启用监控
echo "xpack.monitoring.collection.enabled: true" >> /etc/elasticsearch/elasticsearch.yml
echo "xpack.monitoring.collection.interval: 10s" >> /etc/elasticsearch/elasticsearch.yml

# 重启服务
sudo systemctl restart elasticsearch

2. 监控脚本

#!/bin/bash
# elk-monitor.sh

ELASTICSEARCH_URL="http://localhost:9200"
KIBANA_URL="http://localhost:5601"
LOGSTASH_URL="http://localhost:9600"

# 检查Elasticsearch健康状态
check_elasticsearch() {
    echo "Checking Elasticsearch..."
    health=$(curl -s "$ELASTICSEARCH_URL/_cluster/health" | jq -r '.status')
    if [ "$health" = "green" ] || [ "$health" = "yellow" ]; then
        echo "✓ Elasticsearch is healthy: $health"
    else
        echo "✗ Elasticsearch is unhealthy: $health"
        return 1
    fi
}

# 检查Kibana状态
check_kibana() {
    echo "Checking Kibana..."
    status=$(curl -s "$KIBANA_URL/api/status" | jq -r '.status.overall.state')
    if [ "$status" = "green" ]; then
        echo "✓ Kibana is healthy"
    else
        echo "✗ Kibana is unhealthy: $status"
        return 1
    fi
}

# 检查Logstash状态
check_logstash() {
    echo "Checking Logstash..."
    status=$(curl -s "$LOGSTASH_URL" | jq -r '.status')
    if [ "$status" = "green" ]; then
        echo "✓ Logstash is healthy"
    else
        echo "✗ Logstash is unhealthy: $status"
        return 1
    fi
}

# 主函数
main() {
    echo "ELK Stack Health Check - $(date)"
    echo "================================"
    
    check_elasticsearch
    check_kibana
    check_logstash
    
    echo "================================"
    echo "Health check completed"
}

main

故障排除

1. 常见问题

Elasticsearch启动失败：

# 检查日志
sudo tail -f /var/log/elasticsearch/elk-cluster.log

# 检查JVM内存
jps -v | grep Elasticsearch

# 检查端口占用
netstat -tlnp | grep 9200

# 检查磁盘空间
df -h

# 检查文件描述符
ulimit -n

内存不足问题：

# 调整JVM堆大小
sudo vim /etc/elasticsearch/jvm.options
# 修改 -Xms 和 -Xmx 参数

# 检查系统内存
free -h

# 检查swap使用
swapon --show

2. 诊断工具

# Elasticsearch诊断
curl -X GET "localhost:9200/_cluster/health?pretty"
curl -X GET "localhost:9200/_nodes/stats?pretty"
curl -X GET "localhost:9200/_cat/indices?v"
curl -X GET "localhost:9200/_cat/shards?v"

# 性能分析
curl -X GET "localhost:9200/_nodes/hot_threads"
curl -X GET "localhost:9200/_cluster/pending_tasks"

3. 日志分析

# 实时监控日志
tail -f /var/log/elasticsearch/*.log
tail -f /var/log/kibana/kibana.log
tail -f /var/log/logstash/logstash-plain.log

# 错误日志过滤
grep -i error /var/log/elasticsearch/*.log
grep -i exception /var/log/elasticsearch/*.log

总结

本章详细介绍了ELK Stack的完整安装部署过程，包括：

关键要点

环境准备: 系统要求、Java环境、系统优化
组件安装: Elasticsearch、Kibana、Logstash、Beats
配置管理: 主配置文件、JVM参数、管道配置
集群部署: 多节点配置、负载均衡、高可用
安全配置: 认证授权、SSL/TLS、用户管理
性能调优: 系统优化、内存配置、线程池
监控运维: 健康检查、性能监控、故障排除

最佳实践

合理规划硬件资源和网络架构
严格按照官方文档进行配置
定期备份配置文件和数据
建立完善的监控和告警机制
制定详细的运维操作手册

下一章我们将学习数据收集与处理，包括Beats和Logstash的详细使用方法。

第二章 ELK安装部署与环境配置

目录

环境准备

系统要求

Java环境配置

系统优化

Elasticsearch安装

1. 下载和安装

2. 配置文件

3. 启动和验证

Kibana安装

1. 下载和安装

2. 配置文件

3. 启动和验证

Logstash安装

1. 下载和安装

2. 配置文件

3. JVM配置

Beats安装

1. Filebeat安装

2. Metricbeat安装

集群配置

1. Elasticsearch集群

2. 负载均衡配置

安全配置

1. 启用安全功能

2. 用户和角色管理

性能调优

1. 系统级优化

2. Elasticsearch优化

监控配置

1. 集群监控

2. 监控脚本

故障排除

1. 常见问题

2. 诊断工具

3. 日志分析

总结

关键要点

最佳实践

网站公告

今日签到

热门文章

最新发布