Shipping log files or directories using rsyslog

Alex Berber

Most Unix systems these days come with pre-installed rsyslog, which is a great light weight service to consolidate logs.

You can configure rsyslog to monitor a log file. It can monitor a single log file or directory, and ship them over to Logz.io. In case of directory all first level files will be monitored.

Requirements

The setup assumes that you have a sudo access
Rsyslog version 5.8.0 and above
Allow outgoing TCP traffic to destination port 5000
A common linux distribution

Automatic configuration

Run the following in order to configure your rsyslog daemon to monitor a log file or directory

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t file -a "ovKAKHOqicsjudKFfLTUkITZlXjHdOQK" -l "listener.logz.io" --filepath "PATH_TO_FILE" -tag "TYPE" 
 Copy

PATH_TO_FILE: /path/to/file/or/directory
TYPE: The log type that is being sent. This enables better parsing of your log data.

Run the following in order to configure your rsyslog daemon to monitor a JSON log file

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t file -a "ovKAKHOqicsjudKFfLTUkITZlXjHdOQK" -l "listener.logz.io" --filepath "PATH_TO_FILE" -tag "TYPE" -c json
 Copy

PATH_TO_FILE: /path/to/file/or/directory
TYPE: The log type that is being sent. This enables better parsing of your log data.
Each log should be a single JSON line, that ends with a new line

NOTE: For complete list of known types please see here, if your type is not listed there, please specifiy the type that best identify your data.

Manual Configuration

Step 1 Configure rsyslog file spooling

To ship a log to logz.io, SSH to your Linux server, copy the below code snippet to your terminal window and execute it. The code verifies the working directory exists. In an Ubuntu server, it will set the proper permissions.

sudo mkdir -v /var/spool/rsyslog 
if [ "$(lsb_release -ds | grep Ubuntu)" != "" ]; then
    sudo chown -R syslog:adm /var/spool/rsyslog
fi
 Copy

Step 2 Create a new configuration file for Log.io

sudo vim /etc/rsyslog.d/21-logzio-sample.conf
 Copy

Add the additional configuration

#   -------------------------------------------------------
#        File Logging Directives for Logz.io
#   -------------------------------------------------------
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# File access file:
$InputFileName PATH_TO_FILE
$InputFileTag TYPE:
$InputFileStateFile stat-TYPE
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template logzFormatFileTagName,"[ovKAKHOqicsjudKFfLTUkITZlXjHdOQK] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n"
if $programname == 'TYPE' then @@listener.logz.io:5000;logzFormatFileTagName
if $programname == 'TYPE' then ~
 Copy

In the following sample please replace:

PATH_TO_FILE: /path/to/file/or/directory
TYPE: The log type that is being sent. This enables better parsing of your log data

NOTE: For complete list of known types please see here, if your type is not listed there, please specifiy the type that best identify your data.

Step 3 Restart rsyslog

After editing and saving the file, execute the following command

sudo service rsyslog restart
 Copy

Step 4 Verify that logs are been sent

You can verify that logs are been sent by searching for log file type on Kibana.

Troubleshooting

If you have any trouble with rsyslog you can check our rsyslog troubleshooting guide here.