Step-by-Step Guide to Installing Logstash on Ubuntu 18.04

by

Logstash stands as a premier open-source tool and the world’s leading log analysis platform, adept at collecting, parsing, and storing logs for future utilization. Equipped with an extensive array of plugins and a dynamic template language, Logstash simplifies the transformation of data streams. A member of the ELK stack, Logstash can also operate independently. It excels in extracting data from any source through input plugins, executing diverse data transformations, and dispatching the data to numerous destinations via output plugins.

In this comprehensive guide, we will walk you through the steps to install and configure Logstash on an Ubuntu 18.04 server.

Prerequisites

  • An Ubuntu 18.04 server.
  • Root privileges configured on your server.

Getting Started

Begin by updating your system to the latest package versions. Execute the following commands:

apt-get update -y
apt-get upgrade -y

After updating, restart your system to apply the changes.

Install Java

Ensure Java 8 or Java 11 is installed before proceeding. Use the following command to install Java 8:

apt-get install openjdk-8-jdk -y

Verify the installed Java version with:

java -version

Expected output:

openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-1-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

Install Logstash

Installing Logstash requires Elasticsearch. By default, Elasticsearch is not included in Ubuntu 18.04 repositories, necessitating the addition of its repository.

First, install the requisite package:

apt-get install apt-transport-https -y

Add the Elasticsearch GPG key:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -

Add the Elasticsearch repository:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list

Install Elasticsearch and Logstash:

apt-get install elasticsearch logstash -y

Initialize and enable both services to start on boot:

systemctl start logstash
systemctl enable logstash
systemctl start elasticsearch
systemctl enable elasticsearch

Check Logstash’s operational status:

systemctl status logstash

You should see an output resembling:

? logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
   Active: active (running) since DATE; TIME ago
 Main PID: 3790 (java)
    Tasks: 13 (limit: 1110)
   CGroup: /system.slice/logstash.service
           ??3790 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 

DATE TIME ubuntu systemd[1]: Started logstash.

Elasticsearch listens on port 9200 by default. Verify with:

netstat -ant | grep 9200

Expected output:

tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN     
tcp6       0      0 ::1:9200                :::*                    LISTEN

Configure Logstash

Configuring Logstash involves defining Inputs, Filters, and Outputs. Each serves a unique function using specific Logstash plugins.

Configure Logstash Input

Create a configuration file, beats-input.conf, to receive data from Beats over TCP port 5044:

nano /etc/logstash/conf.d/beats-input.conf

Insert:

input {
  beats {
    port => 5044
  }
}

Save and close the file.

Configure Logstash Filters

Create a filter for processing events from Beats. To capture SSH authentication events, create ssh-auth-filter.conf:

nano /etc/logstash/conf.d/ssh-auth-filter.conf

Insert:

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}\s+%{IPORHOST:dst_host}\s+%{WORD:syslog_program}\[\d+\]:\s+(?\w+\s+password)\s+for\s+%{USER:auth_user}\s+from\s+%{SYSLOGHOST:src_host}.*" }
    add_field => { "activity" => "SSH Logins" }
    add_tag => "linux_auth"
  }
}

Save and close the file.

Configure Logstash Output

To send event data to specific destinations, define output plugins. Configure Logstash to send data to a locally hosted Elasticsearch:

nano /etc/logstash/conf.d/elasticsearch-output.conf

Insert:

output {
   elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "ssh_auth-%{+YYYY.MM}"
   }
  stdout { codec => rubydebug }
}

Save and close the file. Restart Logstash:

systemctl restart logstash

Test Logstash

Confirm Logstash is set up correctly. Verify the configuration with:

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Successful output will resemble:

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[DATE TIME][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[DATE TIME][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[DATE TIME][INFO ][org.reflections.Reflections] Reflections took TIME ms to scan X urls, producing Y keys and Z values 
Configuration OK
[DATE TIME][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Congratulations, you have successfully installed and configured Logstash on Ubuntu 18.04 server.

FAQ

What is Logstash?

Logstash is an open-source data collection engine with real-time pipelining capabilities. It collects, processes, and forwards data.

Is Logstash only available as part of the ELK stack?

While frequently used within the ELK stack (Elasticsearch, Logstash, Kibana), Logstash is versatile and can operate independently.

Can I use Java 11 instead of Java 8?

Yes, Logstash supports both Java 8 and Java 11. Ensure compatibility with other components if using within the ELK stack.

Why do I need Elasticsearch with Logstash?

Elasticsearch acts as a search and analytics engine, enabling the storage and retrieval of log data processed by Logstash.