Elastic Stack, often referred to as the ELK Stack, comprises four essential components: Elasticsearch, Logstash, Kibana, and Beats. Notably, Filebeat, a popular member of this family, efficiently collects, forwards, and centralizes event log data to either Elasticsearch or Logstash for indexing. Filebeat’s modules, such as Apache, Nginx, System, MySQL, and auditd, facilitate simplified visualization of common log formats with just a single command.
In this guide, we will walk through the process of installing and configuring Filebeat on Ubuntu 18.04 to forward event logs, specifically SSH authentication events, to Logstash.
Prerequisites
- An Ubuntu 18.04 server with Elasticsearch, Kibana, and Logstash set up and properly configured.
- Access to a root user account or a user with sudo privileges.
Getting Started
To ensure your system is up to date, run the following commands:
apt-get update -y apt-get upgrade -y
Reboot your system to implement the updates.
Install Filebeat
Because Filebeat is not included in the default Ubuntu 18.04 repository, you need to add the Elastic Stack 7 APT repository:
First, install the required package:
apt-get install apt-transport-https -y
Download and add the Elastic Stack key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
Add the Elastic Stack 7 APT repository:
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list
After adding the repository, update it and install Filebeat:
apt-get update -y apt-get install filebeat -y
Configure Filebeat
By default, Filebeat is set to send event data to Elasticsearch. To configure Filebeat to send data to Logstash, edit the /etc/filebeat/filebeat.yml file:
nano /etc/filebeat/filebeat.yml
Comment out the Elasticsearch output section and uncomment the Logstash output section like this:
#-------------------------- Elasticsearch output ------------------------------ # output.elasticsearch: # Array of hosts to connect to. # hosts: ["localhost:9200"] #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["localhost:5044"]
Enable Filebeat System Module
Filebeat includes numerous modules. You can list them all with:
filebeat modules list
You will see that many modules are disabled by default:
Enabled: Disabled: apache auditd aws [...] system
To enable the system module, which collects and parses logs from your system’s logging service, use:
filebeat modules enable system
Verify the system module is enabled:
filebeat modules list
The output should confirm the module is enabled:
Enabled: system
Configure the system module to read only authentication logs by editing:
nano /etc/filebeat/modules.d/system.yml
Modify the file as follows:
- module: system
# Syslog
syslog:
enabled: false
...
# Authorization logs
auth:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/auth.log"]
Load the Index Template in Elasticsearch
Load the template into Elasticsearch manually by executing:
filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
This command should confirm the setup:
Index setup finished.
Generate and install the index template with:
filebeat export template > filebeat.template.json curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/filebeat-7.0.1 -d@filebeat.template.json
Test Elasticsearch Data Reception
Verify that Elasticsearch is receiving data by running:
curl -X GET localhost:9200/_cat/indices?v
You should see an output similar to:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana_task_manager_1 fpHT_GhXT3i_w_0Ob1bmrA 1 0 2 0 46.1kb 46.1kb yellow open ssh_auth-2019.11 mtyIxhUFTp65WqVoriFvGA 1 1 15154 0 5.7mb 5.7mb [...]
Add Index on Kibana
Log into your Kibana dashboard and navigate to Index Patterns. Follow these steps illustrated in the screenshots:
Click on Create index pattern.
Enter ssh_auth-* as the index pattern and proceed by clicking the Next button.
Select @timestamp and finalize by clicking Create index pattern.
Access the Discover tab to view your data:
Congratulations! You have successfully set up Filebeat to forward event data to Logstash. You can now utilize Kibana dashboards to visualize and analyze your data efficiently.
Frequently Asked Questions
1. Can I use a different version of Ubuntu?
Yes, while this guide is tailored for Ubuntu 18.04, the steps are generally applicable to other Ubuntu versions and Debian-based systems with minor adjustments. Always ensure you refer to the relevant documentation for any specific differences.
2. Why do we need to configure Filebeat to send data to Logstash instead of Elasticsearch?
Logstash offers additional processing capabilities, allowing you to transform, filter, and enrich data before indexing it in Elasticsearch. This is beneficial if your data processing needs are more complex.
3. How do I troubleshoot if Filebeat is not sending logs?
Firstly, ensure that the Filebeat service is running. Check the Filebeat logs located in /var/log/filebeat. Moreover, validate the connection details to Logstash and Elasticsearch are correctly specified in the configuration files.
4. Can I use Filebeat to monitor other log types?
Yes, Filebeat supports various modules for different log types. Refer to the Filebeat documentation and enable the relevant modules for your specific use case.
5. Can I automate these installation steps?
Absolutely. You can use shell scripts, Ansible, Puppet, or other configuration management tools to automate the setup and configuration of Filebeat.