Part 6 - Robustify

In Part One, We talked about hardware
In Part Two, We Ubuntified your MicroSD card.
In Part Three, We got everything set up to my personal liking, but ymmv.
In Part Four, We built CoreDNS!
In Part Five, We deployed and configured CoreDNS

Here in Part Six We make the host more durable →→→

And in Part Seven We add the bits for the eInk display

Part Six

Robustification
- Install stuff
- NFSmount /backup
- lldpd
- sysctls
- logrotate config
- snmpd
- syslog-ng
- Disable IPv6
- observium
- collectd
- prometheus
- ssh keys
- git repo
- cpuspeed governor

🐺🔥⚗️

So here, we’re have more of a smorgasbord of various improvements. Not all of them will be useful to anyone… myself included…
As some of these… improvements… grow, it will make sense to split them out into their own sub-pages.

Package Installs

We need to install some stuff…. Namely

nfsclient packages
lldpd
shell tools
monitoring and telemetry tools
python3
mailutils

apt-get install bwm-ng cpufrequtils cpuinfo cpuset libconfig9 \
libfreetype-dev libio-pty-perl libipc-run-perl libpng-dev \
libpng-tools libsnmp-base libsnmp40 libtime-duration-perl \
libtimedate-perl lldpad lldpd mailutils moreutils nfs-common \
prometheus-node-exporter prometheus-node-exporter-collectors \
python-is-python3 python3-pip smartmontools snmp \
snmp-mibs-downloader sockperf spi-tools tuna tcputils

Apt will ask you to set up postfix.. The nuances and challenges of setting up and maintaining a mail exchanger are myriad and outside the scope of this document.

NFSmount /backup

You may, or may not care to set up an NFS Backup mountpoint. If ya do, click to expand this an see wot I did.. ↕

You cared! How nice!!
Thanks!

In my environment, I have a fileserver separate from most of the infrastructure.
Personally, I find it helpful, to have a durable sharepoint I can store ’things for future me’ and have them reside on completely isolated hardware from the thing in question.

This practice has saved mny ass more than once, not gonna lie. However, it’s not strictly a requirement.

Setting up, and maintaining an NFS server becomes nuanced as dependencies increase. Guidance for this is outside the scope of this document.

I create distinct lenses into the storage for each context within my network. such that NFS traffic to a specific segment of the environment needn’t traverse a firewall. so, each subnet has its’ own address for the local nfs server..

Your environment may have different needs, and you may choose to deal with this differently. YMMV.

/etc/fstab:

LABEL=writable                                 /              ext4 discard,errors=remount-ro               0 1
LABEL=system-boot                              /boot/firmware vfat defaults                                0 1
#10.18.40.23:/Backups/Infrastructure/DNSServer /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.18.42.23:/Backups/Infrastructure/DNSServer /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.18.40.23:/Backups/Infrastructure/DNSServer /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.0.2.23:/Backups/Infrastructure/DNSServer   /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0

lldpd

I found a few sources of “documentation” that were somewhat helpful… So I’ll include ¹ them ² here ³…
However,

You might not give a shit about lldpd.... Click here if you do though. ↕

So, for the most part, you just sorta install it and move on, but there’s definitely some interesting stuff to explore.

for i in `ls /sys/class/net/ | grep 'eth\|ens\|eno'` ;
do echo "enabling lldp for interface: $i" ;
lldptool set-lldp -i $i adminStatus=rxtx ;
lldptool -T -i $i -V sysName enableTx=yes;
lldptool -T -i $i -V portDesc enableTx=yes ;
lldptool -T -i $i -V sysDesc enableTx=yes;
lldptool -T -i $i -V sysCap enableTx=yes;
lldptool -T -i $i -V mngAddr enableTx=yes;
configure med location address country US
done
IP=`ip addr show eth0 |awk '/inet/ {print $2}'|sed -e 's/\/.*$//'`; lldptool -T -i eth0 -V mngAddripv4=${IP}

lldptool configure med location address county Travis country US country-subdivision Texas building Barn room Barn

lldptool configure med location address county Travis country US country-subdivision Texas building Haus room Office

Sysctls

I spent a fair bit of time dorking around with the information on the calomel.org⁴ site, specifically their network performance page⁵. There’s a lot of nifty stuff there.

However, you might not care too much about tweaking sysctls. Click here if you want to see my settings as of 09/2022. ↕

This is a starting point tuning similar to what I’ve set up on proxmox. As time goes on I’ll tune this more apropriately for dns servers.

/etc/sysctl.d/9931_WPL_SYSCTLS-0922.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
net.core.netdev_max_backlog=13888
net.core.rmem_max=16777216
net.ipv4.tcp_rmem=4096 131072 16777216
net.core.wmem_max=16777216
net.ipv4.tcp_wmem=4096 131072 16777216
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1
net.ipv4.tcp_fack=0
net.ipv4.route.min_adv_mss=536
net.ipv4.tcp_base_mss=1460
net.ipv4.tcp_min_snd_mss=536

Journald

Journald is the new-ish systemd logging mechanism.. It’s got some neat configuration knobs.

The most useful is to push syslogs elsewhere, and set storage to volatile. this offloads a large about of unnecessary writes.

Click here to see my journald.conf file. ↕

/etc/systemd/journald.conf:

[Journal]
Storage=volatile
ForwardToSyslog=yes
MaxLevelSyslog=info
MaxLevelStore=info
Compress=no
SystemMaxFileSize=2M
SystemMaxFiles=200
SystemMaxUse=400M
RuntimeMaxFileSize=2M
RuntimeMaxUse=400M
RuntimeMaxFiles=200
MaxFileSec=1week

SNMP

You might knock it….

Hey! SNMPD has been around forever. it’s old and busted. Where’s the new hotness monitoring shit?

And you’d be right. SNMPD HAS been around forever. v1 in 88⁶. and v2 in 91⁷.
Here’s the thing tho…
It still works

click here to see my installation of snmpd and the related accoutrement ↕

so with snmpd apt-get install snmp-mibs-downloader snmpd

agentAddress  udp:161
sysLocation    ['30.3295110,-97.7560740']
sysName        coredns-01
sysContact     Me <coredns@wolfspaw.com>
sysObjectID 1.3.6.1.4.1.50536.3.1
#sysServices    72
sysServices    76
rocommunity "w01f5paw" default
view   default included   .1
view   default included   .1.3.6.1.2.1
#view   default included   .1.3.6.1.2.1.25.1

proc lldpd
proc lldpad
proc sshd
proc coredns


disk       /     10000
disk       /var  5%
includeAllDisks  10%

load   12 10 5
#  Arbitrary extension commands
#
#extend-sh test3   /bin/sh /tmp/shtest
master          agentx
extend .1.3.6.1.4.1.2021.7890.2 hardware /bin/cat /sys/devices/virtual/dmi/id/product_name
extend .1.3.6.1.4.1.2021.7890.3 vendor   /bin/cat /sys/devices/virtual/dmi/id/sys_vendor
extend .1.3.6.1.4.1.2021.7890.4 serial   /bin/cat /sys/devices/virtual/dmi/id/product_serial
extend uptime /bin/cat /proc/uptime
extend .1.3.6.1.4.1.2021.7890.1 distro /usr/local/bin/distro
# This line enables Observium's ifAlias description injection
pass_persist .1.3.6.1.2.1.31.1.1.1.18 /usr/local/bin/ifAlias_persist

# Allow Systems Management Data Engine SNMP to connect to snmpd using SMUX
smuxpeer .1.3.6.1.4.1.674.10892.1

timesyncd

Expand ↕

/etc/systemd/timesyncd.conf

1
2
3
4
5
6
[Time]
NTP=timey.wolfspyre.io wimey.wolfspyre.io wobly.wolfspyre.io tick.wolfspyre.io tock.wolfspyre.io
#FallbackNTP=ntp.ubuntu.com
RootDistanceMaxSec=4
PollIntervalMinSec=16
PollIntervalMaxSec=64

logrotate

Expand ↕

Git Prompt / Completion

Expand ↕

rsyslog

Expand ↕

ca certificates

You probably don’t care about internal CA Certificates

Expand ↕

disable ipv6

place this in /etc/sysctl.d/9933_WPL_DISABLE_ipv6.conf

1
2
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

🐺🔥⚗️

⬅⃖⃖ Back to Deploy & Config

…Continue to Extra Credit ➡⃗⃗