The Hoof & Paw
DocsCategoriesTagsView the current conditions from the WolfspyreLabs WeatherstationToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeBack to homepage

Part 6 - Robustify

  • In Part One, We talked about hardware
  • In Part Two, We Ubuntified your MicroSD card.
  • In Part Three, We got everything set up to my personal liking, but ymmv.
  • In Part Four, We built CoreDNS!
  • In Part Five, We deployed and configured CoreDNS
Here in Part Six We make the host more durable β†’β†’β†’
  • And in Part Seven We add the bits for the eInk display

Part Six

🐺πŸ”₯βš—οΈ

So here, we’re have more of a smorgasbord of various improvements. Not all of them will be useful to anyone… myself included
As some of these… improvements… grow, it will make sense to split them out into their own sub-pages.

Package Installs

We need to install some stuff…. Namely

  • nfsclient packages
  • lldpd
  • shell tools
  • monitoring and telemetry tools
  • python3
  • mailutils
apt-get install bwm-ng cpufrequtils cpuinfo cpuset libconfig9 \
libfreetype-dev libio-pty-perl libipc-run-perl libpng-dev \
libpng-tools libsnmp-base libsnmp40 libtime-duration-perl \
libtimedate-perl lldpad lldpd mailutils moreutils nfs-common \
prometheus-node-exporter prometheus-node-exporter-collectors \
python-is-python3 python3-pip smartmontools snmp \
snmp-mibs-downloader sockperf spi-tools tuna tcputils  

Apt will ask you to set up postfix.. The nuances and challenges of setting up and maintaining a mail exchanger are myriad and outside the scope of this document.

NFSmount /backup

You cared! How nice!!
Thanks!

In my environment, I have a fileserver separate from most of the infrastructure.
Personally, I find it helpful, to have a durable sharepoint I can store ’things for future me’ and have them reside on completely isolated hardware from the thing in question.

This practice has saved mny ass more than once, not gonna lie. However, it’s not strictly a requirement.

Setting up, and maintaining an NFS server becomes nuanced as dependencies increase. Guidance for this is outside the scope of this document.

I create distinct lenses into the storage for each context within my network. such that NFS traffic to a specific segment of the environment needn’t traverse a firewall. so, each subnet has its’ own address for the local nfs server..

Your environment may have different needs, and you may choose to deal with this differently. YMMV.

/etc/fstab:

LABEL=writable                                 /              ext4 discard,errors=remount-ro               0 1
LABEL=system-boot                              /boot/firmware vfat defaults                                0 1
#10.18.40.23:/Backups/Infrastructure/DNSServer /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.18.42.23:/Backups/Infrastructure/DNSServer /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.18.40.23:/Backups/Infrastructure/DNSServer /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.0.2.23:/Backups/Infrastructure/DNSServer   /backup        nfs  rw,relatime,rsize=1048576,wsize=1048576 0 0

lldpd

I found a few sources of “documentation” that were somewhat helpful… So I’ll include 1 them 2 here 3
However,

So, for the most part, you just sorta install it and move on, but there’s definitely some interesting stuff to explore.

for i in `ls /sys/class/net/ | grep 'eth\|ens\|eno'` ;
do echo "enabling lldp for interface: $i" ;
lldptool set-lldp -i $i adminStatus=rxtx ;
lldptool -T -i $i -V sysName enableTx=yes;
lldptool -T -i $i -V portDesc enableTx=yes ;
lldptool -T -i $i -V sysDesc enableTx=yes;
lldptool -T -i $i -V sysCap enableTx=yes;
lldptool -T -i $i -V mngAddr enableTx=yes;
configure med location address country US
done
IP=`ip addr show eth0 |awk '/inet/ {print $2}'|sed -e 's/\/.*$//'`; lldptool -T -i eth0 -V mngAddripv4=${IP}

lldptool configure med location address county Travis country US country-subdivision Texas building Barn room Barn

lldptool configure med location address county Travis country US country-subdivision Texas building Haus room Office

Sysctls

I spent a fair bit of time dorking around with the information on the calomel.org4 site, specifically their network performance page5. There’s a lot of nifty stuff there.

This is a starting point tuning similar to what I’ve set up on proxmox. As time goes on I’ll tune this more apropriately for dns servers.

/etc/sysctl.d/9931_WPL_SYSCTLS-0922.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
net.core.netdev_max_backlog=13888
net.core.rmem_max=16777216
net.ipv4.tcp_rmem=4096 131072 16777216
net.core.wmem_max=16777216
net.ipv4.tcp_wmem=4096 131072 16777216
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1
net.ipv4.tcp_fack=0
net.ipv4.route.min_adv_mss=536
net.ipv4.tcp_base_mss=1460
net.ipv4.tcp_min_snd_mss=536

Journald

Journald is the new-ish systemd logging mechanism.. It’s got some neat configuration knobs.

The most useful is to push syslogs elsewhere, and set storage to volatile. this offloads a large about of unnecessary writes.

/etc/systemd/journald.conf:

[Journal]
Storage=volatile
ForwardToSyslog=yes
MaxLevelSyslog=info
MaxLevelStore=info
Compress=no
SystemMaxFileSize=2M
SystemMaxFiles=200
SystemMaxUse=400M
RuntimeMaxFileSize=2M
RuntimeMaxUse=400M
RuntimeMaxFiles=200
MaxFileSec=1week

SNMP

You might knock it….

Hey! SNMPD has been around forever. it’s old and busted. Where’s the new hotness monitoring shit?

And you’d be right. SNMPD HAS been around forever. v1 in 886. and v2 in 917.
Here’s the thing tho…
It still works

so with snmpd apt-get install snmp-mibs-downloader snmpd

agentAddress  udp:161
sysLocation    ['30.3295110,-97.7560740']
sysName        coredns-01
sysContact     Me 
sysObjectID 1.3.6.1.4.1.50536.3.1
#sysServices    72
sysServices    76
rocommunity "w01f5paw" default
view   default included   .1
view   default included   .1.3.6.1.2.1
#view   default included   .1.3.6.1.2.1.25.1

proc lldpd
proc lldpad
proc sshd
proc coredns


disk       /     10000
disk       /var  5%
includeAllDisks  10%

load   12 10 5
#  Arbitrary extension commands
#
#extend-sh test3   /bin/sh /tmp/shtest
master          agentx
extend .1.3.6.1.4.1.2021.7890.2 hardware /bin/cat /sys/devices/virtual/dmi/id/product_name
extend .1.3.6.1.4.1.2021.7890.3 vendor   /bin/cat /sys/devices/virtual/dmi/id/sys_vendor
extend .1.3.6.1.4.1.2021.7890.4 serial   /bin/cat /sys/devices/virtual/dmi/id/product_serial
extend uptime /bin/cat /proc/uptime
extend .1.3.6.1.4.1.2021.7890.1 distro /usr/local/bin/distro
# This line enables Observium's ifAlias description injection
pass_persist .1.3.6.1.2.1.31.1.1.1.18 /usr/local/bin/ifAlias_persist

# Allow Systems Management Data Engine SNMP to connect to snmpd using SMUX
smuxpeer .1.3.6.1.4.1.674.10892.1

timesyncd

/etc/systemd/timesyncd.conf

1
2
3
4
5
6
[Time]
NTP=timey.wolfspyre.io wimey.wolfspyre.io wobly.wolfspyre.io tick.wolfspyre.io tock.wolfspyre.io
#FallbackNTP=ntp.ubuntu.com
RootDistanceMaxSec=4
PollIntervalMinSec=16
PollIntervalMaxSec=64

logrotate

Git Prompt / Completion

rsyslog

ca certificates

You probably don’t care about internal CA Certificates

disable ipv6

place this in /etc/sysctl.d/9933_WPL_DISABLE_ipv6.conf

1
2
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
🐺πŸ”₯βš—οΈ