Fairly recently I made some notes for a setup of Elastic Stack on a network of CentOS 6 machines. I found it relatively involved so thought it was worth sharing.
On the main log processing server
Oracle Java 8 needs to be installed.
Import RPM key:
rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
In /etc/yum.repos.d/elasticsearch.repo:
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Install Elasticsearch:
yum install elasticsearch
Add the following to /etc/init.d/elasticsearch:
# Configure Java environment
JAVA_HOME=/usr/local/java [or /usr/local/jdk8 if needed]
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME PATH
Start the service:
service elasticsearch start
chkconfig elasticsearch on
In /etc/yum.repos.d/kibana.repo:
[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Install Kibana:
yum install kibana
Configure Kibana:
In /etc/kibana/kibana.yml:
server.host: "0.0.0.0"
Start the service:
service kibana start
chkconfig kibana on
In /etc/yum.repos.d/logstash.repo:
[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Install Logstash:
yum install logstash
If needed, change Java executable path in /etc/logstash/startup.options then run /usr/share/logstash/bin/system-install.
Add to /etc/init/logstash.conf:
env JAVA_HOME=/usr/local/java
env PATH=/usr/local/java/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
Add syslog source in /etc/logstash/conf.d/syslog.conf:
input {
file {
path => [ "/var/log/messages" ]
type => "syslog"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
Add filter in /etc/logstash/conf.d/filebeat.conf:
input {
beats {
port => 5044
}
}
filter {
if [fields][log_type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
If necessary, fix permissions on /var/log/messages to make it readable as needed, then start logstash.
Results should be visible at this URL (insert name/IP as appropriate for your network): http://NAME OR IP OF LOG PROCESSING SERVER:5601
To tail logs for information and problems:
cd /var/log
tail -F messages elasticsearch/*.log kibana/* logstash/logstash-plain.log
On other servers sending log data to the main server
Import RPM key:
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
In /etc/yum.repos.d/elastic.repo:
[elastic-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Install Filebeat:
yum install filebeat
Modify /etc/filebeat/filebeat.yml as follows:
filebeat.prospectors:
- input_type: log
paths:
- /var/log/messages
fields:
log_type: syslog
output.logstash:
hosts: ["NAME OR IP OF LOG PROCESSING SERVER:5044"]
Start service:
chkconfig --add filebeat
service filebeat start
To tail logs for information and problems:
tail -F /var/log/filebeat/filebeat