In order make Logstash send logs to Logsene, you need to configure it with the Elasticsearch plugin to output to the Logsene endpoint, while specifying:
logsene-receiver.sematext.com as the host
- 80 or 443 as the port
- your Logsene App token as the index name
- http as the protocol
- ssl as true for SSL/HTTPS
Tailing a File
To send the contents of a file, you'd configure it like this:
input { file { path => "/var/log/apache.log" start_position => "beginning" # this will also send existing contents the first time you start Logstash add_field => { "source" => "apache" } # add a source field, for easier filtering } } output { elasticsearch { host => "logsene-receiver.sematext.com" ssl => true # (requires Logstash 1.5+) if you do not want to use SSL comment this out and change port to 80 port => 443 # or 80 is you disabled SSL with ssl => false protocol => http # this should NOT be changed to HTTPS even if you enable SSL with ssl => true index => "LOGSENE_APP_TOKEN_GOES_HERE" manage_template => false # Logsene will manage templates for you } }
To get started quickly, you can simply download Logstash, unpack it, save the above configuration into a file (e.g., /etc/logstash/conf.d/logsene.conf), then start Logstash:
bin/logstash -f /tmp/logstash.conf
After it starts, logs from that file will flow to Logsene and you should be able to start searching them.
Parsing Unstructured Data
You may want to extract metrics from your logs, to do various analysis tasks. For example, you can make a pie chart in Kibana that shows you how often clients get 200 response codes, how often 500 and so on.
With Logstash, you can parse logs by using the grok filter. For example, you can enhance the previous configuration to parse your Apache combined log format by changing it to the following:
input { file { path => "/var/log/apache.log" add_field => { "source" => "apache" } # add a source field, for easier filtering start_position => "beginning" } } filter { if [source] == "apache" { # only try to parse if logs are from the "apache" source grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } } } output { elasticsearch { host => "logsene-receiver.sematext.com" port => 80 index => "LOGSENE_APP_TOKEN_GOES_HERE" protocol => http manage_template => false } }
Tagging Specific Logs
Logstash populates a number of special fields, such as host and @timestamp. You can also configure it to add one or more tags to logs matching certain criteria. This is useful when you want to quickly identify a special kind of logs. For example, you could tag events that come from the apache log and contain the word "error" as "apache errors".
To achieve this, you can use the mutate filter to add the tags, and wrap it in a conditional that matches those specific logs:
filter { if [source] == "apache" and [message] =~ "error" { mutate { add_field => { "tags" => ["apache errors" ] } } } }