filebeat收集日志到es

前言

Filebeat版本：8.18
配置参考：https://www.elastic.co/guide/en/beats/filebeat/8.18/multiline-examples.html
下载地址：https://www.elastic.co/downloads/beats/filebeat

一、下载filebeat

打开：https://www.elastic.co/downloads/beats/filebeat
下载对应系统、对于版本的filebeat

二、修改配置

在/root/filebeat-8.18.2-linux-x86_64下新增bztcFilebeat.yml

vim bztcFilebeat.yml

将以下配置粘贴到yml中：

filebeat.inputs:
  - type: filestream
    id: bztc-gateway
    enabled: true
    paths:
      - /root/bztc-gateway/bztc-log/bztc-gateway/all.log
    parsers:
      - multiline:
          type: pattern
          pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
          negate: true
          match: after
    fields:
      service_name: "gateway"
    fields_under_root: true
    ignore_older: 72h
    processors:
      - add_host_metadata:
          netinfo.enabled: true
      - script:
          lang: javascript
          id: extract_logtime
          source: >
            function process(event) {
              var str = event.Get("message");
              if (str != null) {
                var time = str.split(" ").slice(0, 2).join(" ");
                event.Put("logtime", time);
              }
            }
      - timestamp:
          field: logtime
          timezone: Asia/Shanghai
          layouts:
            - '2006-01-02 15:04:05.000'
            - '2006-01-02 15:04:05'

  - type: filestream
    id: bztc-notify
    enabled: true
    paths:
      - /root/bztc-notify/bztc-log/bztc-notify/all.log
    parsers:
      - multiline:
          type: pattern
          pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
          negate: true
          match: after
    fields:
      service_name: "notify"
    fields_under_root: true
    ignore_older: 72h
    processors:
      - add_host_metadata:
          netinfo.enabled: true
      - script:
          lang: javascript
          id: extract_logtime
          source: >
            function process(event) {
              var str = event.Get("message");
              if (str != null) {
                var time = str.split(" ").slice(0, 2).join(" ");
                event.Put("logtime", time);
              }
            }
      - timestamp:
          field: logtime
          timezone: Asia/Shanghai
          layouts:
            - '2006-01-02 15:04:05.000'
            - '2006-01-02 15:04:05'

output.elasticsearch:
  hosts: ["https://192.168.0.133:9200", "https://192.168.0.134:9200", "https://192.168.0.135:9200"]
  username: elastic
  password: elastic
  ssl.certificate_authorities: ["./certs/ca.crt"]
  index: "bztc-log-%{[service_name]}"

setup.ilm.enabled: false

setup.template.enabled: true
setup.template.name: "bztc-log-template"
setup.template.pattern: "bztc-log-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

其中的配置说明参考：https://www.elastic.co/guide/en/beats/filebeat/8.18/multiline-examples.html

三、运行

运行以下命令：

cd /root/filebeat-8.18.2-linux-x86_64
./filebeat -c bztcFilebeat.yml

运行成功后截图：

如有报错，需具体问题具体分析。

四、kibana中查看

索引：

数据流：

索引模板：

查看日志：

总结

filebeat收集日志到es