In this guide we will discuss about Supported Inputs in Logstash. Logstash supports a huge range of logs from different sources. It is working with famous sources as explained below.
Collect Logs from Metrics
System events and other time activities are recorded in metrics. Logstash can access the log from system metrics and process them using filters. This helps to show the user the live feed of the events in a customized manner. Metrics are flushed according to the flush_interval setting of metrics filter and by default; it is set to 5 seconds.
We are tracking the test metrics generated by Logstash, by gathering and analyzing the events running through Logstash and showing the live feed on the command prompt.
logstash.conf
This configuration contains a generator plugin, which is offered by Logstash for test metrics and set the type setting to “generated” for parsing. In the filtering phase, we are only processing the lines with a generated type by using the ‘if’ statement. Then, the metrics plugin counts the field specified in meter settings. The metrics plugin flushes the count after every 5 seconds specified in the flush_interval.
Lastly, output the filter events to a standard output like command prompt using the codec plugin for formatting. The Codec plugin is using [events][rate_1m] value to output the per second events in a 1-minute sliding window.
input { generator { type => "generated" } } filter { if [type] == "generated" { metrics { meter => "events" add_tag => "metric" } } } output { # only emit events with the 'metric' tag if "metric" in [tags] { stdout { codec => line { format => "rate: %{[events][rate_1m]}" } } }
Run Logstash
We can run Logstash by using the following command.
>logsaths –f logstash.conf
stdout (command prompt)
rate: 1308.4 rate: 1308.4 rate: 1368.654529135342 rate: 1416.4796003951449 rate: 1464.974293984808 rate: 1523.3119444107458 rate: 1564.1602979542715 rate: 1610.6496496890895 rate: 1645.2184750334154 rate: 1688.7768007612485 rate: 1714.652283095914 rate: 1752.5150680019278 rate: 1785.9432934744932 rate: 1806.912181962126 rate: 1836.0070454626025 rate: 1849.5669494173826 rate: 1871.3814756851832 rate: 1883.3443123790712 rate: 1906.4879113216743 rate: 1925.9420717997118 rate: 1934.166137658981 rate: 1954.3176526556897 rate: 1957.0107444542625
Collect Logs from the Web Server
Web servers generate a large number of logs regarding user access and errors. Logstash helps to extract the logs from different servers using input plugins and stash them in a centralized location.
We are extracting the data from the stderr logs of the local Apache Tomcat Server and stashing it in the output.log.
logstash.conf
This Logstash configuration file directs Logstash to read apache error logs and add a tag named “apache-error”. We can simply send it to the output.log using the file output plugin.
input { file { path => "C:/Program Files/Apache Software Foundation/Tomcat 7.0 /logs/*stderr*" type => "apache-error" } } output { file { path => "C:/tpwork/logstash/bin/log/output.log" } }
Run Logstash
We can run Logstash by using the following command.
>Logstash –f Logstash.conf
Sample of Input log
This is the sample stderr log, which generates when the server events occur in Apache Tomcat.
C:\Program Files\Apache Software Foundation\Tomcat 7.0\logs\ tomcat7-stderr.2016-12-25.log
Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-9999"] Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-bio-8009"] Dec 25, 2016 7:05:14 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 823 ms
output.log
{ "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/ tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z", "@version":"1","host":"Dell-PC", "message":"Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start\r", "type":"apache-error","tags":[] } { "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/ tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z", "@version":"1","host":"Dell-PC", "message":"INFO: Starting ProtocolHandler [ \"ajp-bio-8009\"]\r","type":"apache-error","tags":[] } { "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/ tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z", "@version":"1","host":"Dell-PC", "message":"Dec 25, 2016 7:05:14 PM org.apache.catalina.startup.Catalina start\r", "type":"apache-error","tags":[] } { "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/ tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z", "@version":"1","host":"Dell-PC", "message":"INFO: Server startup in 823 ms\r","type":"apache-error","tags":[] }
Collect Logs from Data sources
To start with, let us understand how to Configure MySQL for logging. Add the following lines in my.ini file of the MySQL database server under [mysqld].
In windows, it is present inside the installation directory of MySQL, which is in −
C:\wamp\bin\mysql\mysql5.7.11
In UNIX, you can find it in – /etc/mysql/my.cnf
general_log_file = "C:/wamp/logs/queries.log" general_log = 1
logstash.conf
In this config file, file plugin is used to read the MySQL log and write it to the ouput.log.
input { file { path => "C:/wamp/logs/queries.log" } } output { file { path => "C:/tpwork/logstash/bin/log/output.log" } }
queries.log
This is the log generated by queries executed in the MySQL database.
2016-12-25T13:05:36.854619Z 2 Query select * from test1_users 2016-12-25T13:05:51.822475Z 2 Query select count(*) from users 2016-12-25T13:05:59.998942Z 2 Query select count(*) from test1_users
output.log
{ "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:05:37.905Z", "@version":"1","host":"Dell-PC", "message":"2016-12-25T13:05:36.854619Z 2 Query\tselect * from test1_users", "tags":[] } { "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:05:51.938Z", "@version":"1","host":"Dell-PC", "message":"2016-12-25T13:05:51.822475Z 2 Query\tselect count(*) from users", "tags":[] } { "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:06:00.950Z", "@version":"1","host":"Dell-PC", "message":"2016-12-25T13:05:59.998942Z 2 Query\tselect count(*) from test1_users", "tags":[] }
Next Topic : Click Here
Pingback: Logstash - Collecting Logs | Adglob Infosystem Pvt Ltd