Base Setup:
Setup Instance:
Update Security Group
Port 5601: Kibana
Port 9200: Elasticsearch
Building ELK:
Install Elasticsearch:
SSH into Instance:
Commands to install Elastic Search
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get update
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update
sudo apt-get install elasticsearch
Now that we have installed Elasticsearch we must change some configs
It is a good best practice to bind Elasticsearch to the private IP (like 172.31.something):
sudo nano /etc/elasticsearch/elasticsearch.yml
Now to start Elasticsearch
sudo service elasticsearch start
Test Elasticsearch
curl http://<Private IP>:9200
Install Logstash:
sudo apt-get install default-jre
Verify the java version with the command below
Now install logstash
sudo apt-get install logstash
Create data pipeline:
mkdir /logstash
cd /logstash
# pull down sample data
sudo chown -R logstash /logstash
sudo chgrp -R logstash /logstash
Create Logstash Configuration File:
sudo nano /etc/logstash/conf.d/apache-01.conf
To test config
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/apache-01.conf
It can take a minute or so to run - but should get a configuration OK message
Start Logstash
sudo service logstash start
Test if it's working
curl http://<Private IP>:9200/_cat/indices?v
Install Kibana:
sudo apt-get install kibana
Edit config
sudo nano /etc/kibana/kibana.yml
Change the lines below
server.port: 5601
server.host: '<YourPrivateIP>'
elasticsearch.hosts: ["http://<YourPrivateIP>:9200"]
Start Kibana
sudo service kibana start
Test Kibana
Add an Index Pattern to display to Logstash Index
Go to Stack Management → Kibana -> Index Patterns - select “Create Index Pattern”
Use Kibana to query data
