filebeat 6.4.3 采集 nginx日志

标签：RoCE   内容   无法   temp   nginx 模块   pack   ast   packages   集群安装   

nginx 日志格式规范：

将日志配置组合添加至nginx主配置文件；

     log_format  eslog  ‘$remote_addr - $remote_user [$time_local] "$request" ‘
                      ‘$status $body_bytes_sent "$http_referer" ‘
                      ‘"$http_user_agent" "$http_x_forwarded_for"‘;

ES集群安装插件

集群收集nginx日志需要安装 ingest-user-agent ，ingest-geoip ，安装完成之后需要重启es 服务，否则数据无法正常录入es集群。

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
/usr/share/elasticsearch/bin/elasticsearch-plugin list

升级java至1.8

yum install java-1.8 -y

安装filebeat

导入key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

创建repo 文件

more lostash.repo

[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装

yum install filebeat -y

修改filebeat主配置文件

more /etc/filebeat/filebeat.yml

filebeat.config:
prospectors:
path: ${path.config}/prospectors.d/.yml
reload.enabled: false
modules:
path: /etc/filebeat/modules.d/.yml
reload.enabled: false

output.elasticsearch:
hosts: [‘10.2.3.30:9200‘]

setup.kibana:
host: "10.2.3.13:5601"

配置filebeat nginx 模块

Access 日志：/data/nginxlog/eslog/es-access.*
Error 日志：/data/nginxlog/eslog/es-error.*

more /etc/filebeat/modules.d/nginx.yml 
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/nginxlog/eslog/es-access.*"]
  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/nginxlog/eslog/es-error.*"]

添加启动

systemctl enable filebeat

开启nginx 模块

cd /etc/filebeat

filebeat modules enable nginx

filebeat modules list

初始化环境

此操作会自动导入filebeat模板和nginx dashboard 到es 集群：

Set up the initial environment:
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations

./filebeat setup -e

启动服务

systemctl start filebeat

启动服务报错

内容如下：

Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern:
  error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: returned 400 to import file: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"Request Timeout after 30000ms"}

2018-10-31T16:35:45.659+0900    INFO    kibana/client.go:113    Kibana url: http://10.2.3.13:5601
2018-10-31T16:37:15.664+0900    ERROR   instance/beat.go:743    Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern:
  error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: fail to execute the HTTP POST request: Post http://10.2.3.13:5601/api/kibana/dashboards/import?force=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response: 

处理办法：

登陆kibana 进入Dev tools 删除

删除旧模板

    #DELETE _template/filebeat-6.X.X 

删除旧数据

    #DELETE filebeat-6.4.2-*

重新导入模板：

cd /etc/filebeat/
filebeat setup
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations

重启服务：

systemctl restart filebeat

Dashboard 效果：

filebeat 6.4.3 采集 nginx日志

标签：RoCE   内容   无法   temp   nginx 模块   pack   ast   packages   集群安装   

原文地址：http://blog.51cto.com/michaelkang/2317953