IMPORTANT
The latest version of rsyslog is 8.x. Even the latest releases of the most popular Linux distributions such as Ubuntu, Debian, RHEL, CentOS, etc. come with very old versions of rsyslog (e.g. 5.x and 7.x). To avoid issues with old versions of rsyslog, please upgrade it to 8.x. We strongly recommend you do this. This is very easy to do and rsyslog developers provide the needed packages and install commands at http://www.rsyslog.com/downloads/download-other/.
Overview
rsyslog is a modern syslog daemon focused on performance. Logsene gives you lots of ways to forward your logs with rsyslog:
- UDP
- TCP (you can also encrypt logs over TLS)
- RELP (uses application-level acknowledgement for increased reliability over plain TCP)
- HTTP / HTTPS over the Elasticsearch API
You can also send JSON over syslog if you need support for structured data.
There are 3 steps for configuring your rsyslog for Logsene:
- Choose one or more inputs. For example, set rsyslog to listen for local logs, remote logs over TCP and so on
- Choose a protocol and an authentication method. For UDP, TCP, TLS and RELP you can authorize your public IP. However, we strongly recommend using the Logsene application's token, which works with all supported protocols
- Configure the output. Based on the chosen protocol and authentication method, you'll have to configure the appropriate output plugin to send logs in the desired format
Configuring Inputs
First, make rsyslog receive logs you want to send to Logsene. The most common input module is imuxsock, which will take logs from your local /dev/log socket. To start listening on local logs, add this line at the beginning of your /etc/rsyslog.conf:
$ModLoad imuxsock
Tailing Files
You can also have rsyslog tail files, listen for syslog messages over TCP, UDP, RELP, pick up messages from the journal and more.
To tail a file, load the file input module, and optionally decide how often to look for changes. Then, for every file, specify its path and a few more parameters, like whether you want multiline support:
If you have issues with logrotate or other utilities that truncate or move the files you monitor, upgrade rsyslog to version 8.1.5 or later and the problems should go away. The file input module gets inotify support, and you also have a new configuration format, that's easier to maintain:
Protocol and Authentication
To forward logs, you can use HTTP/HTTPS and authenticate by using your Logsene application token (recommended!). Alternatively, you can use UDP, TCP/TLS or RELP and authenticate either by using the Logsene application token, or by authorizing your public IP from the Logsene UI.
HTTP/HTTPS via the Elasticsearch API
The recommended method is to use the Elasticsearch API and send logs over HTTP or HTTPS. This will give you maximum flexibility, reliability and encryption if you need it. This requires rsyslog 6.4.0 or later, and the installation of the Elasticsearch output module. HTTPS is supported in rsyslog 8.2.0 or later (see info about rsyslog update above). To enable the Elasticsearch output module, install the rsyslog-elasticsearch package or use --enable-elasticsearch when compiling from sources.
Configuration
Before forwarding via the Elasticsearch API, define a template in /etc/rsyslog.conf that makes your syslog messages JSON:
UDP, TCP, TLS or RELP
You can send RFC-3164 or RFC-5424 compatible syslog to Logsene via any of the following protocols:
- UDP: this is the fire-and-forget protocol that doesn't guarantee any reliability but is also the fastest and simplest implementation
- TCP: this provides better reliability than TCP, though messages might still be lost from the system buffers if the connection breaks and rsyslog is restarted
- TLS: RFC-5425 syslog over TLS is supported
- RELP: this is a reliable protocol, because it supports application-level acknowledgments
With all these protocols, you can either authenticate with your Logsene application token, or by registering your public IP.
Requirements
UDP and TCP support is built in to rsyslog. For TLS, you need to install the gtls driver, typically provided by the rsyslog-gnutls package, or by adding --enable-gnutls when compiling from sources. For RELP, you'll need the RELP output module, normally provided by the rsyslog-relp package or by adding --enable-relp when compiling from sources.
Configuration
To use the Logsene application token, you'll first have to get it from your list of Logsene applications. Then, in your rsyslog.conf, define a template in /etc/rsyslog.conf that forwards your syslog messages in CEE-formatted JSON over syslog, where you put your token in the logsene-app-token field:
$template LogseneFormat,"<%PRI%>%TIMEREPORTED:::date-rfc3339% %HOSTNAME% %syslogtag%@cee: {\"logsene-app-token\": \"LOGSENE_APP_TOKEN_GOES_HERE\", \"message\": \"%msg:::json%\"}\n"
If you're using version 7 or later, you can use the new configuration format to define the template. It's more verbose, but also easier to add new fields if you want to:
If you prefer authorizing IPs to using tokens, you don't need to make any configuration change for this step. Instead, go to the Logsene UI and authorize the public IP (or IPs) from which you send your logs.
Configuring Outputs
The last step is about configuring the output module to send your logs to Logsene. This depends on your chosen protocol.
HTTP / HTTPS via Omelasticsearch
To send your logs over HTTP, load the Elasticsearch output module and then point it to logsene-receiver.sematext.com on port 80. Make sure you replace LOGSENE_APP_TOKEN_GOES_HERE with your actual token:
If you want to encrypt logs over HTTPS, you'll have to change serverport to "443" and add usehttps="on".
UDP
If you're using the Logsene application token for authentication, specify the LogseneFormat template in your action line. The host you'll connect to is logsene-receiver-syslog.sematext.com:
*.* @logsene-receiver-syslog.sematext.com;LogseneFormat
If you've authorized your public IPs, the RFC-5424 predefined template will do:
*.* @logsene-receiver-syslog.sematext.com;RSYSLOG_SyslogProtocol23Format
TCP
With TCP, the action line looks similar to UDP, except you'll have two @ signs. This is for when your using the Logsene application token:
*.* @@logsene-receiver-syslog.sematext.com;LogseneFormat
This is for when you authorize IPs:
*.* @@logsene-receiver-syslog.sematext.com;RSYSLOG_SyslogProtocol23Format
TLS
To set up syslog over TLS, first set up the certificates:
Then, configure the TLS driver:
Finally, add the action line, which is the same as the TCP one, except you'll use port 10514. For authorizing with the Logsene application token:
*.* @@logsene-receiver-syslog.sematext.com:10514;LogseneFormat
And for authorizing the IPs:
*.* @@logsene-receiver-syslog.sematext.com:10514;RSYSLOG_SyslogProtocol23Format
If you prefer the new configuration format, you can find the whole TLS configuration below:
RELP
To forward via RELP, load the RELP output module and then point it to logsene-receiver-syslog.sematext.com on port 20514. As with TCP or UDP, specify the LogseneFormat template for authorizing with your Logsene application token:
$ModLoad omrelp *.* :omrelp:logsene-receiver-syslog.sematext.com:20514;LogseneFormat
And the RFC-5424 predefined template for IP-based authorization:
$ModLoad omrelp *.* :omrelp:logsene-receiver-syslog.sematext.com:20514;RSYSLOG_SyslogProtocol23Format
Tag Your Logs
From your syslog messages, Logsene will populate a number of special fields, such as the source and host. You can also configure rsyslog to add one or more tags to logs matching certain criteria. This is useful when you want to quickly identify a special kind of logs. For example, you could tag events that come to the "kernel" facility with a severity/level of "error" as both "kernel errors" and "urgent issues".
To achieve this, define a template similar to the ones described above, where you'd add a tags field. Then, you'd use a conditional to match those messages and send them to Logsene using the newly defined template:
$template kernelErrors,"<%PRI%>%TIMEREPORTED:::date-rfc3339% %HOSTNAME% %syslogtag%@cee: {\"logsene-app-token\": \"LOGSENE-APP-TOKEN-GOES-HERE\", \"message\": \"%msg:::json%\", \"tags\":[\"kernel errors\", \"urgent issues\"]}\n" |
Notice the &~ statement - this prevents rsyslog from sending matched events twice (once with tags and once without). Make sure you place these conditionals before your main Logsene action (the one starting with *.* ).