The Hoof & Paw
DocsCategoriesTagsView the current conditions from the WolfspyreLabs WeatherstationToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeBack to homepage

Sending Syslog to Loki

3 minutes to read Wolf Noble   Tech   Grafana Logs Loki Promtail Rsyslog Tech Troubleshooting

Getting Syslog messages into Loki

So, now that we’ve figured out Where our logs were going1… lets push the rest of the logs into the stack

Ship them to promtail with some buffering

The rsyslog site’s guide for shipping logs2 recommends the following configuration for nodes:

*.*  action(type="omfwd" target="192.0.2.2" port="10514"
            protocol="tcp" action.resumeRetryCount="100" 
            queue.type="linkedList" queue.size="10000")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# this is the simplest forwarding action:
*.* action(type="omfwd" target="192.0.2.1" port="10514" protocol="tcp")
# it is equivalent to the following obsolete legacy format line:
*.* @@192.0.2.1:10514 
# do NOT use this any longer!

# Note: if the remote system is unreachable, processing will
# block here and discard messages after a while

# so a better use is
*.*  action(type="omfwd" target="192.0.2.2" port="10514" protocol="tcp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000")
# this will de-couple the sending from the other logging actions,
# and prevent delays when the remote system is not reachable. Also,
# it will try to connect 100 times before it discards messages as
# undeliverable.
# the rest below is more or less a plain vanilla rsyslog.conf as 
# many distros ship it - it's more for your reference...
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none      /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                    /var/log/secure
# Log all the mail messages in one place.
mail.*                                        /var/log/maillog
# Log cron stuff
cron.*                                        /var/log/cron
# Everybody gets emergency messages
*.emerg                                       :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                /var/log/spooler
# Save boot messages also to boot.log
local7.*                                      /var/log/boot.log

So, why not just make a few simple template files to push out to nodetypes

/etc/rsyslog.d/wpl-gitlablogs.confLang: syslog
*.*  action(type="omfwd"
       protocol="tcp" target="elky.nfo.wolfspyre.io" port="5520"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")
# udp/tcp 5520 = syslog-ng listener for Gitlab
# tcp 15520 = promtail listener  for Gitlab

/etc/rsyslog.d/wpl-corednslog.confLang: syslog
*.*  action(type="omfwd"
       protocol="tcp" target="elky.nfo.wolfspyre.io" port="5553"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")
# udp/tcp 5553 = syslog-ng listener for CoreDNS
# tcp 15553 = promtail listener  for CoreDNS
/etc/rsyslog.d/wpl-prom-syslog.confLang: syslog
*.*  action(type="omfwd"
       protocol="tcp" target="elky.nfo.wolfspyre.io" port="16514"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")
# udp/tcp 6514 = syslog-ng listener for syslog
# tcp 16514 = promtail listener  for syslog

  1. It REALLY helps to point promtail at the right loki instance if you wish loki to ingest aforementioned logs. sigh ↩︎

  2. https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/ ↩︎