Setup ELK(ElasticSearch、Logstash、Kibana) Environment

Posted on | In | Page View

Setup ELK(ElasticSearch、Logstash、Kibana) Environment

  • 此文章描述如何建立ELK運行環境，以及如何啟動與使用。
  • ElasticSearch的安裝可以參考此篇文章

Basic Info

  • OS: CentOS 7.3
  • Nginx Version: 1.8.0
  • ElasticSearch Version: 5.5.0
  • Logstash Version: 5.5.0
  • Kibana Version: 5.5.0

Logstash Installation

About Logstash Version

Install Logstash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@localhost ~]# cd /opt/software
[root@localhost software]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.tar.gz
[root@localhost software]# tar -zxvf logstash-5.5.0.tar.gz
[root@localhost software]# chown -R datacenter:datacenter logstash-5.5.0
[root@localhost software]# vim /etc/profile
export LOGSTASH_HOME="/opt/software/logstash-5.5.0"
export PATH="$LOGSTASH_HOME/bin:$PATH"
[root@localhost software]# source /etc/profile
[root@localhost software]# cd logstash-5.5.0
[root@localhost logstash-5.5.0]# ll
total 180
drwxr-xr-x 2 datacenter datacenter    205 Jan 11 00:25 bin
-rw-r--r-- 1 datacenter datacenter 111573 Jun 30  2017 CHANGELOG.md
drwxr-xr-x 2 datacenter datacenter     93 Jan 11 00:25 config
-rw-r--r-- 1 datacenter datacenter   2249 Jun 30  2017 CONTRIBUTORS
drwxr-xr-x 2 datacenter datacenter      6 Jun 30  2017 data
-rw-r--r-- 1 datacenter datacenter   3874 Jun 30  2017 Gemfile
-rw-r--r-- 1 datacenter datacenter  21427 Jun 30  2017 Gemfile.jruby-1.9.lock
drwxr-xr-x 5 datacenter datacenter     65 Jan 11 00:25 lib
-rw-r--r-- 1 datacenter datacenter    589 Jun 30  2017 LICENSE
drwxr-xr-x 4 datacenter datacenter     84 Jan 11 00:25 logstash-core
drwxr-xr-x 3 datacenter datacenter     57 Jan 11 00:25 logstash-core-plugin-api
drwxr-xr-x 3 datacenter datacenter     40 Jan 11 00:25 modules
-rw-r--r-- 1 datacenter datacenter  29345 Jun 30  2017 NOTICE.TXT
drwxr-xr-x 3 datacenter datacenter     30 Jan 11 00:25 tools
drwxr-xr-x 4 datacenter datacenter     33 Jan 11 00:25 vendor

Configure Logstash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
## 新增config/logstash-nginx-access-log.conf
[root@localhost logstash-5.5.0]# vim config/logstash-nginx-access-log.conf
    file {
        path => ["/usr/local/nginx/logs/access.log"]
        type => "nginx_access"
        start_position => "beginning"
    }
}
filter {
  grok {
    match => {
      "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
    }
  }
  date {
    match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }}
output {
  elasticsearch {
        hosts => ["10.103.67.84:9200"]
        index => "logstash-nginx-access-log"
    }
}

Kibana Installation

About Logstash Version

Install Kibana

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@localhost ~]# cd /opt/software
[root@localhost software]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-linux-x86_64.tar.gz
[root@localhost software]# tar -zxvf kibana-5.5.0-linux-x86_64.tar.gz
[root@localhost software]# chown -R datacenter:datacenter kibana-5.5.0-linux-x86_64
[root@localhost software]# vim /etc/profile
export KIBANA_HOME="/opt/software/kibana-5.5.0-linux-x86_64"
export PATH="$KIBANA_HOME/bin::$PATH"
[root@localhost software]# source /etc/profile
[root@localhost software]# cd kibana-5.5.0-linux-x86_64/
[root@localhost kibana-5.5.0-linux-x86_64]# ll
total 836
drwxr-xr-x   2 datacenter datacenter     41 Jun 30  2017 bin
drwxrwxr-x   2 datacenter datacenter     24 Jun 30  2017 config
drwxrwxr-x   2 datacenter datacenter      6 Jun 30  2017 data
-rw-rw-r--   1 datacenter datacenter    562 Jun 30  2017 LICENSE.txt
drwxrwxr-x   6 datacenter datacenter    108 Jun 30  2017 node
drwxrwxr-x 620 datacenter datacenter  20480 Jun 30  2017 node_modules
-rw-rw-r--   1 datacenter datacenter 800210 Jun 30  2017 NOTICE.txt
drwxrwxr-x   3 datacenter datacenter     45 Jun 30  2017 optimize
-rw-rw-r--   1 datacenter datacenter    721 Jun 30  2017 package.json
drwxrwxr-x   2 datacenter datacenter      6 Jun 30  2017 plugins
-rw-rw-r--   1 datacenter datacenter   4909 Jun 30  2017 README.txt
drwxr-xr-x  11 datacenter datacenter    139 Jun 30  2017 src
drwxrwxr-x   5 datacenter datacenter     52 Jun 30  2017 ui_framework
drwxr-xr-x   2 datacenter datacenter    309 Jun 30  2017 webpackShims

Configure Kibana

1
2
3
4
5
6
7
8
[root@localhost kibana-5.5.0-linux-x86_64]# vim config/kibana.yml
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "10.103.67.84"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://10.103.67.84:9200"

Start ELK Process

Set Nginx Log Format and Start Nginx

  • 使用Nginx作為Logstash的輸入，Nginx環境安裝可以參考此篇文章
  • 將Nginx輸出Log的格式指定爲Logstash可以接受的格式
    • 對應logstash-nginx-access-log.conf中的match
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@localhost ~]# vim $NGINX_HOME/conf/nginx.conf
## 就下列注釋的設定開啓
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  logs/access.log  main;
## 使用root啓動Nginx
[root@localhost ~]# nginx
## 使用瀏覽器訪問 http://[IP]
[root@localhost ~]# tail -F $NGINX_HOME/logs/access.log
10.50.129.66 - - [11/Jan/2018:01:31:14 -0800] "GET / HTTP/1.0" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36" "10.103.67.80"
10.50.129.66 - - [11/Jan/2018:01:32:27 -0800] "GET / HTTP/1.0" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36" "10.103.67.80"

Start ElasticSearch

1
2
3
4
5
6
7
8
9
10
11
12
13
## 使用一般user啓動ElasticSearch
[datacenter@localhost ~]$ elasticsearch
## ...
[2018-01-11T00:43:54,299][INFO ][o.e.n.Node               ] [es_001_168] initialized
[2018-01-11T00:43:54,300][INFO ][o.e.n.Node               ] [es_001_168] starting ...
[2018-01-11T00:43:54,605][INFO ][o.e.t.TransportService   ] [es_001_168] publish_address {10.103.67.84:9300}, bound_addresses {10.103.67.84:9300}
[2018-01-11T00:43:54,621][INFO ][o.e.b.BootstrapChecks    ] [es_001_168] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-11T00:43:57,669][INFO ][o.e.c.s.ClusterService   ] [es_001_168] new_master {es_001_168}{1_CXoUhGTIOrcYEoFg-3lA}{CLazd0x-STKi4gEktJLmvA}{10.103.67.84}{10.103.67.84:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-11T00:43:57,733][INFO ][o.e.h.n.Netty4HttpServerTransport] [es_001_168] publish_address {10.103.67.84:9200}, bound_addresses {10.103.67.84:9200}
[2018-01-11T00:43:57,733][INFO ][o.e.n.Node               ] [es_001_168] started
[2018-01-11T00:43:57,740][INFO ][o.e.g.GatewayService     ] [es_001_168] recovered [0] indices into cluster_state
[2018-01-11T00:45:19,924][INFO ][o.e.c.m.MetaDataCreateIndexService] [es_001_168] [logstash-nginx-access-log] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]
[2018-01-11T00:45:20,174][INFO ][o.e.c.m.MetaDataMappingService] [es_001_168] [logstash-nginx-access-log/x6oK1CEwTRu2SM5jNSBjdg] create_mapping [nginx_access]

Start Logstash

1
2
3
4
5
6
7
8
9
10
11
12
13
## 使用一般user啓動Logstash
[datacenter@localhost ~]$ logstash -f $LOGSTASH_HOME/config/logstash-nginx-access-log.conf
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /opt/software/logstash-5.5.0/logs which is now configured via log4j2.properties
[2018-01-11T00:52:41,667][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.103.67.84:9200/]}}
[2018-01-11T00:52:41,673][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.103.67.84:9200/, :path=>"/"}
[2018-01-11T00:52:41,806][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x178f59c9>}
[2018-01-11T00:52:41,807][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-01-11T00:52:41,888][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-01-11T00:52:41,899][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x205413cf>]}
[2018-01-11T00:52:41,967][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-01-11T00:52:42,174][INFO ][logstash.pipeline        ] Pipeline main started
[2018-01-11T00:52:42,241][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Start Kibana

1
2
3
4
5
6
7
8
9
10
11
[datacenter@localhost ~]$ kibana
  log   [09:02:44.070] [info][status][plugin:kibana@5.5.0] Status changed from uninitialized to green - Ready
  log   [09:02:44.126] [info][status][plugin:elasticsearch@5.5.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:02:44.144] [info][status][plugin:console@5.5.0] Status changed from uninitialized to green - Ready
  log   [09:02:44.158] [info][status][plugin:metrics@5.5.0] Status changed from uninitialized to green - Ready
  log   [09:02:44.354] [info][status][plugin:timelion@5.5.0] Status changed from uninitialized to green - Ready
  log   [09:02:44.358] [info][listening] Server running at http://10.103.67.84:5601
  log   [09:02:44.359] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
  log   [09:02:49.400] [info][status][plugin:elasticsearch@5.5.0] Status changed from yellow to yellow - No existing Kibana index found
  log   [09:02:49.870] [info][status][plugin:elasticsearch@5.5.0] Status changed from yellow to green - Kibana index ready
  log   [09:02:49.872] [info][status][ui settings] Status changed from yellow to green - Ready

Kibana Web UI

  • Kibana initialization web

  • Kibana set pattern web

  • Kibana index pattern web