Skip to main content

Elastic Beats on pfSense : Installation and configuration

Summary

Though in many cases syslog is preferred to transport the pfSense logs to external system, Elastic beats provides quite a niche way to send the logs while modelling the data alongside. This makes it ready-made to send to ElasticSearch directly and get ready-made outcomes like SIEM, performance etc.

Pre-reqs

  • A build server (preferably Ubuntu or Fedora) with internet connectivity
  • shell access to pfsense server
  • Basic knowledge of Elastic Stack (filebeat.yml configurations etc)
  • Ensure connectivity is allowed from pfsense machine to your Elastic Stack receiver

Setup Summary

  • Connectivity tests
  • Install dependencies in build server (vagrant, virtualbox, gmake, go etc)
  • Download Elastic Beats source
  • Make elastic Beats package for FreeBSD
  • Copy binary packages to pfsense server
  • Configure Beats to send to destination
  • Configure ElasticSearch to view the data

Installation Steps

Connectivity tests

 Logon to pfsense server via Shell 
ssh root@192.168.1.1
Password for root@pf.localdomain:pfSense - Netgate Device ID: 1a6323512345bf9e165d2
*** Welcome to pfSense 2.4.5-RELEASE (amd64) on pf ***
WAN (wan) -> re0 -> v4: 192.168.1.3/24
LAN (lan) -> re1 -> v4: 192.168.2.1/24
0) Logout (SSH only) 9) pfTop
1) Assign Interfaces 10) Filter Logs
2) Set interface(s) IP address 11) Restart webConfigurator
3) Reset webConfigurator password 12) PHP shell + pfSense tools
4) Reset to factory defaults 13) Update from console
5) Reboot system 14) Disable Secure Shell (sshd)
6) Halt system 15) Restore recent configuration
7) Ping host 16) Restart PHP-FPM
8) Shell
Enter an option: 8

Test connectivity to your ElasticSearch from pfsense

[root@pf.localdomain]/root: ssh -v -p 9200 192.168.1.10  # This is elasticSearch Server 
OpenSSH_7.5p1, OpenSSL 1.0.2u-freebsd 20 Dec 2019debug1: Reading configuration data /etc/ssh/ssh_configdebug1: Connecting to 192.168.1.10 [192.168.1.10] port 9200.debug1: Connection established.
Test Internet connectivity from build Server (just ping github.com)
[root@buildserver]/tmp: ping github.com
PING github.com (140.82.118.4):
56 data bytes64 bytes from 140.82.118.4: icmp_seq=0 ttl=57 time=19.443 ms
64 bytes from 140.82.118.4: icmp_seq=1 ttl=57 time=19.348

Install dependencies in build server

  • Ensure VirtualBox is installed. (or follow link)
  • Ensure Vagrant is installed
[root@buildserver]/tmp: mkdir /tmp/freebsd 
[root@buildserver]/tmp: cd /tmp/freebsd/
[root@buildserver]/tmp/freebsd: wget https://releases.hashicorp.com/vagrant/2.2.9/vagrant_2.2.9_x86_64.deb [root@buildserver]/tmp/freebsd: apt install ./vagrant_2.2.9_x86_64.deb

Create Vagrant file for FreeBSD

  • Update the vagrant file
 [[root@buildserver]/tmp/freebsd: vi Vagrantfile 
# FreeBSD Vagrant
Vagrant.configure("2") do |config|
config.vm.box = "freebsd/FreeBSD-11.2-RELEASE"
config.vm.guest = :freebsd
config.ssh.shell = "sh"
config.vm.network "private_network", type: "dhcp"
config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
config.vm.box_version = "2018.06.22"
end
Bring up the container and ssh to it
[root@buildserver]/tmp/freebsd: vagrant up
[root@buildserver]/tmp/freebsd: vagrant ssh
Now the FreeBSD container is ready. Just logon to container and switch to root within it and install the dependencies
[root@freeBSDContainer]/tmp/: pkg install git gmake go bashUpdating pfSense-core repository catalogue...pfSense-core repository is up to date.Updating pfSense repository catalogue.....The following 13 package(s) will be affected (of 0 checked):
exit the shell and relogin
Download Elastic Beats source
[root@freeBSDContainer]/root: go get github.com/elastic/beats
package github.com/elastic/beats: build constraints exclude all Go files in /root/go/src/github.com/elastic/beats

Please take of the location /root/go/src/github.com/elastic/beats

Make elastic Beats package for FreeBSD

[root@freeBSDContainer]/tmp: cd /root/go/src/github.com/elastic/beats 
[root@freeBSDContainer]/tmp: git checkout v7.8.0
[root@freeBSDContainer]/root/go/src/github.com/elastic/beats/filebeat: gmake
[root@freeBSDContainer]/root/go/src/github.com/elastic/beats/filebeat: file filebeat
[root@freeBSDContainer]/root/go/src/github.com/elastic/beats/filebeat: ./filebeat version
[root@freeBSDContainer]/root/go/src/github.com/elastic/beats/metricbeat: ./metricbeat version

Copy binary packages to pfsense server

  • Using scp or other methods, copy the file to pfsense server. Ideally its good to keep everything under /opt/beats/
  • configure Beats to send to destination in the relevant *beat.yml (eg filebeat.yml)
  • Ideally you shouldn't start filebeat as root, but since this is for test purposes, we will do so. Else you need to create user and setup it correctly with least privileges
[[root@pf.localdomain]/root/go/src/github.com/elastic/beats/filebeat: vi filebeat.yml 
# ============================== Filebeat Config START ==============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 120s
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
hosts: ["192.168.1.10:9200"]
protocol: "http"
#api_key: "id:api_key"
username: "elastic"
password: "whateverpassword"
# ============================== Filebeat Config END ==============================
Ensure the relevant modules in filebeat are enabled
[root@pf.localdomain]/root/go/src/github.com/elastic/beats/filebeat: mv modules.d/auditd.yml.disabled modules.d/auditd.yml
Start filebeat
[root@pf.localdomain]/root/go/src/github.com/elastic/beats/filebeat: ./filebeat -e
  • check for any errors. otherwise you will see data coming to ElasticSearch cluster in few minutes.
  • Configure ElasticSearch to view the data (this includes indices, index-patterns, index-templates)
 

Popular posts from this blog

Create your own Passport Photo using GIMP

This tutorial is for semi-techies who knows a bit of GIMP (image editing).   This tutorial is for UK style passport photo ( 45mm x 35 mm ) which is widely used in UK, Australia, New Zealand, India etc.  This is a quick and easy process and one can create Passport photos at home If you are non-technical, use this link   .  If you want to create United States (USA) Passport photo or Overseas Citizen of India (OCI) photo, please follow this link How to Make your own Passport Photo - Prerequisite GIMP - One of the best image editing tools and its completely Free USB stick or any memory device to store and take to nearby shop A quality Digital camera Local Shops where you can print. Normally it costs (£0.15 or 25 US cents) to print 8 photos Steps (Video Tutorial attached blow of this page) Ask one of your colleague to take a photo  of you with a light background. Further details of how to take a photo  yourself       Take multiple pictures so that you can choose from th

Syslog Standards: A simple Comparison between RFC3164 & RFC5424

Syslog Standards: A simple Comparison between RFC3164 (old format) & RFC5424 (new format) Though syslog standards have been for quite long time, lot of people still doesn't understand the formats in detail. The original standard document is quite lengthy to read and purpose of this article is to explain with examples Some of things you might need to understand The RFC standards can be used in any syslog daemon (syslog-ng, rsyslog etc.) Always try to capture the data in these standards. Especially when you have log aggregation like Splunk or Elastic, these templates are built-in which makes your life simple. Syslog can work with both UDP & TCP  Link to the documents the original BSD format ( RFC3164 ) the “new” format ( RFC5424 ) RFC3164 (the old format) RFC3164 originated from combining multiple implementations (Year 2001)