ELK + Pfsense 1/2

Why this tutorial?

Welcome to the my ELK + Pfsense tutorial. You have probably searched on a search engine like Google or Bing for tutorials which are describing Elasticsearch and how to use that with your Pfsense. I did the same in 2016/2017 and yes I found some tutorials and you will find also some tutorials from 2017/2016/2015/2014. So why I'm going to write my own tutorial? All tutorials I found online (probably I know all of them;) are incomplete. They're not just incomplete in every tutorial there are missing steps. Wrong configuration files, old versions of ELK and/or Pfsense. Often they are based on a different distribution like Ubuntu and they're not based on a minimal installation of the operating system. This means no tutorial you find online will work out of the box. I have done this job and have opened several threads on discuss.elastic.co . I have updated the system more than 4 times and I always got it to work. The information and configuration files you will find in this tutorial is based from several people / websites / blogs / posts from user in online forums. Why this whole procedure -> because it's fun and if it's working you just like it. My brother was the reason for all of this, he discovered back in 2013/2014 some screenshot from Kibana with dashboards like in a movie.

Last I want to warn you because there is no guarantee for anything. A first hint from my side: don't waste your time with writing e-mails or anything to other people's tutorial. Most of the time you won't get an answer. I have written to all of them ;=).

Introductions

What do you need do get all of this working together. Some steps are already required and I'm sorry at the moment I can't cover them in this tutorial. Most of them are not so complicated and are standard installations so there you will find online an official documentation of it.

Hardware

  • - Hypervisor (esXi V.6.5 or higher, Microsoft HYP-V 2016), It will probably work on V.5.5 or even V.5.0 but I have never tested that, the same for HYP-V Windows 2012 / R2. If you don't use a hypervisor you can run CentOS on physical machine as well.
  • - Mouse and keyboard
  • - LAN and Internet
  • - Second physical machine with a browser or a virtual machine
  • - Time and energy! ( I spent around 400 hours into ELK + Pfsense, you can probably do it in 10 hours or less)

Software

  • - Pfsense installation out of the box V.2.4.3 (This is not part of this tutorial) - It's also possible to use your current running Pfsense
  • - CentOS 7 (64 Bit) minimal installation (NO GUI) - (Yes, it will probably work on a full installation)
  • - Internet for downloading some tools and ELK on your CentOS 7 (64 Bit) machine

 If you have all the requirements then you're ready to start. It is recommended that you're familiar with command line interfaces (putty and ssh or shell). So I assume that you know how to connect via putty to your CentOS 7 machine.

 

Installation Java V.8 Update 161

First login to your CentOS machine via CLI and put the following line:

If you don't have wget you neet install it first via the first line and after that you're able to do the wget command.

yum install wget

wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jre-8u161-linux-x64.rpm"

Now you have downloaded the Java package and you need to install it.

sudo yum localinstall jre-8u161-linux-x64.rpm

After you have done the installation we will check if Java 8 is installed.

java -version

Java installation

Installation Elasticsearch

First we will import the Elasticsearch GPG-KEY.

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Now we're downloading for the first time Elasticsearch.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.rpm

Our next step is the installation of Elasticseach.

rpm -ivh elasticsearch-6.2.3.rpm

Now you have to change your directory.

cd /etc/elasticsearch/

We're now editing the configuration of Elasticsearch.

vi elasticsearch.yml

You have to change a couple of things in it. Go to to the following lines and do what I wrote in the brackets.

bootstrap.memory_lock: true (Comment out this means, remove the #)
network.host: localhost (Comment out this means, remove the #, change to localhost)
http.port: 9200 (Comment out this means, remove the #)

Now we need to Limit Memlock (This step I'm not sure if it's necessary). We do this directly in the service.

vi /usr/lib/systemd/system/elasticsearch.service

Enter a new line and add the following option:

LimitMEMLOCK=infinity

Open the Sysconfig of Elasticsearch.

vi /etc/sysconfig/elasticsearch

The next line you have to uncomment as above and just remove #

MAX_LOCKED_MEMORY=unlimited

Max Locked Memory

Now we start the service new and reload the deamon as well as the automatic startup on boot.

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

If you made everything correct and then you now need Net-Tools which is not included in the minimal installation of CentOS 7. You can get Net-Tools by entering the next command.

sudo yum install net-tools

Now we can check if Elasticsearch is running and up.

curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'
curl -XGET 'localhost:9200/?pretty'

On the next side I will explain how we install Kibana and Logstash. You're already on a good way, just keep going.... If everything worked fine you will get an image like that. It's important that "You Know, for Search" is written. Then you now Elasticsearch is running.

Elasticsearch running

Back to Top