Blog

This is an open-source, centralized, events and logging manager. It is a part of the ELK (ElasticSearch, Logstash, Kibana) stack. In this chapter, we will understand the basics of Logstash, its features, and the various components it has.

Audience

This chapter is designed for software professionals who want to learn the basics of Logstash and its programming concepts in simple and easy steps. It describes the components and functions of Logstash with suitable examples.

Prerequisites

The readers are expected to have a basic understanding of Ruby, JSON, and web technologies. Additionally it will be helpful for the readers to be familiar with Logging Techniques and Regex patterns.

Introduction

This is a tool based on the filter/pipes patterns for gathering, processing and generating the logs or events. It helps in centralizing and making real time analysis of logs and events from different sources.

It is written on JRuby programming language that runs on the JVM, hence you can run Logstash on different platforms. It collects different types of data like Logs, Packets, Events, Transactions, Timestamp Data, etc., from almost every type of source. The data source can be Social data, E-commerce, News articles, CRM, Game data, Web trends, Financial data, Internet of Things, Mobile devices, etc.

Logstash General Features

The general features of Logstash are as follows −

• It can collect data from different sources and send to multiple destinations.
• This can handle all types of logging data like Apache Logs, Windows Event Logs, Data over Network Protocols, Data from Standard Input and many more.
• This can also handle http requests and response data.
• It provides a variety of filters, which helps the user to find more meaning in the data by parsing and transforming it.
• It can also be used for handling sensors data in internet of things.
• This is open source and available under the Apache license version 2.0.

Logstash Key Concepts

The key concepts of Logstash are as follows −

Event Object

It is the main object in Logstash, which encapsulates the data flow in the Logstash pipeline. It uses this object to store the input data and add extra fields created during the filter stage.

It offers an Event API to developers to manipulate events. In this chapter, this event is referred with various names like Logging Data Event, Log Event, Log Data, Input Log Data, Output Log Data, etc.

Pipeline

It comprises of data flow stages in Logstash from input to output. The input data is entered in the pipeline and is processed in the form of an event. Then sends to an output destination in the user or end system’s desirable format.

Input

This is the first stage in the Logstash pipeline, which is used to get the data in Logstash for further processing. It offers various plugins to get data from different platforms. Some of the most commonly used plugins are – File, Syslog, Redis and Beats.

Filter

This is the middle stage of Logstash, where the actual processing of events take place. A developer can use pre-defined Regex Patterns by Logstash to create sequences for differentiating between the fields in the events and criteria for accepted input events.

It offers various plugins to help the developer to parse and transform the events into a desirable structure. Some of the most commonly used filter plugins are – Grok, Mutate, Drop, Clone and Geoip.

Output

This is the last stage in the Logstash pipeline, where the output events can be formatted into the structure required by the destination systems. Lastly, it sends the output event after complete processing to the destination by using plugins. Some of the most commonly used plugins are – Elasticsearch, File, Graphite, Statsd, etc.

Advantages

The following points explain the various advantages of Logstash.

• It offers regex pattern sequences to identify and parse the various fields in any input event.
• Logstash supports a variety of web servers and data sources for extracting logging data.
• It provides multiple plugins to parse and transform the logging data into any user desirable format.
• This is centralized, which makes it easy to process and collect data from different servers.
• It supports many databases, network protocols and other services as a destination source for the logging events.
• It uses the HTTP protocol, which enables the user to upgrade Elasticsearch versions without having to upgrade Logstash in a lock step.

Disadvantages

The following points explain the various disadvantages of Logstash.

• It uses http, which negatively affects the processing of the logging data.
• Working with Logstash can sometimes be a little complex, as it needs a good understanding and analysis of the input logging data.
• Filter plugins are not generic, so, the user may need to find the correct sequence of patterns to avoid error in parsing.

In the next chapter, we will understand what the ELK Stack is and how it helps Logstash.

Next Topic : Click Here