I noticed that from my OSX desktop, and most of the hosts here, I could download assets significantly faster from El Internetto™ than I could on from my TrueNAS Scale box.
IPC Socket Buffer: the maximum combined socket buffer size, in bytes, defined
by SO_SNDBUF and SO_RCVBUF. kern.ipc.maxsockbuf is also used to define the
window scaling factor (wscale in tcpdump) our server will advertise. The
window scaling factor is defined as the maximum volume of data allowed in
transit before the recieving server is required to send an ACK packet
(acknowledgment) to the sending server. FreeBSD’s default maxsockbuf value is
two(2) megabytes which corresponds to a window scaling factor (wscale) of
six(6) allowing the remote sender to transmit up to 2^6 x 65,535 bytes =
4,194,240 bytes (4MB) in flight, on the network before requiring an ACK
packet from our server. In order to support the throughput of modern, long
fat networks (LFN) with variable latency we suggest increasing the maximum
socket buffer to at least 16MB if the system has enough RAM. “netstat -m”
displays the amount of network buffers used. Increase kern.ipc.maxsockbuf if
the counters for “mbufs denied” or “mbufs delayed” are greater than zero(0).
https://en.wikipedia.org/wiki/TCP_window_scale_optionhttps://en.wikipedia.org/wiki/Bandwidth-delay_product
increasing the number of packets able to be processed in an interrupt is advisable. The default 0 indicates 16 frames (less than 24kB)
man iflib for more info
kern.ipc.shm_use_phys=1
kern.msgbuf_show_timestamp=1
net.inet.ip.portrange.randomtime=5
net.inet.tcp.blackhole=2
net.inet.tcp.fast_finwait2_recycle=1 # recycle FIN/WAIT states quickly, helps against DoS, but may cause false RST (default 0)
net.inet.tcp.fastopen.client_enable=0 # disable TCP Fast Open client side, enforce three way TCP handshake (default 1, enabled)
net.inet.tcp.fastopen.server_enable=0 # disable TCP Fast Open server side, enforce three way TCP handshake (default 0)
net.inet.tcp.finwait2_timeout=1000 # TCP FIN_WAIT_2 timeout waiting for client FIN packet before state close (default 60000, 60 sec)
net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
net.inet.tcp.keepcnt=2 # amount of tcp keep alive probe failures before socket is forced closed (default 8)
net.inet.tcp.keepidle=62000 # time before starting tcp keep alive probes on an idle, TCP connection (default 7200000, 7200 secs)
net.inet.tcp.keepinit=5000 # tcp keep alive client reply timeout (default 75000, 75 secs)
net.inet.tcp.msl=2500 # Maximum Segment Lifetime, time the connection spends in TIME_WAIT state (default 30000, 2*MSL = 60 sec)
net.inet.tcp.path_mtu_discovery=1 # disable for mtu=1500 as most paths drop ICMP type 3 packets, but keep enabled for mtu=9000 (default 1)
net.inet.udp.blackhole=1 # drop udp packets destined for closed sockets (default 0)
net.inet.udp.recvspace=1048576 # UDP receive space, HTTP/3 webserver, "netstat -sn -p udp" and increase if full socket buffers (default 42080)
#security.bsd.hardlink_check_gid=1 # unprivileged processes may not create hard links to files owned by other groups, DISABLE for mailman (default 0)
#security.bsd.hardlink_check_uid=1 # unprivileged processes may not create hard links to files owned by other users, DISABLE for mailman (default 0)
security.bsd.see_other_gids=0 # groups only see their own processes. root can see all (default 1)
security.bsd.see_other_uids=0 # users only see their own processes. root can see all (default 1)
security.bsd.stack_guard_page=1 # insert a stack guard page ahead of growable segments, stack smashing protection (SSP) (default 0)
security.bsd.unprivileged_proc_debug=0 # unprivileged processes may not use process debugging (default 1)
security.bsd.unprivileged_read_msgbuf=0 # unprivileged processes may not read the kernel message buffer (default 1)
net.link.ifqmaxlen #50 -> 2048 per https://redmine.pfsense.org/issues/10311
Things I tried
pkg install devcpu-data-intel-20220510
iovctl
ixl driver
hw.ixl.rx_itr
:The RX interrupt rate value, set to 62 (124 usec) by default.
hw.ixl.tx_itr
:The TX interrupt rate value, set to 122 (244 usec) by default.
hw.ixl.i2c_access_method
: Access method that driver will use for I2C read and writes via sysctl(8) or verbose ifconfig(8) information display:
0 - best available method
1 - bit bang via I2CPARAMS register
2 - register read/write via I2CCMD register
3 - Use Admin Queue command (default best)
Using the Admin Queue is only supported on 710 devices with FW version 1.7 or newer. Set to 0 by default.
hw.ixl.enable_tx_fc_filter
: Filter out packets with Ethertype 0x8808 from being sent out by non-adapter sources. This prevents (potentially untrusted) software or iavf(4) devices from sending out flow control packets and creating a DoS (Denial of Service) event. Enabled by default.
hw.ixl.enable_head_writeback
When the driver is finding the last TX descriptor processed by the hardware, use a value written to memory by the hardware instead of scanning the descriptor ring for completed descriptors. Enabled by default; disable to mimic the TX behavior found in ixgbe(4).
SYSCTL PROCEDURES
dev.ixl.#.fc
Sets the 802.3x flow control mode that the adapter will advertise on the link. The negotiated flow control setting can be viewed in the interface’s media field if ifconfig(8)
- 0 Disables flow control
- 1 is RX
- 2 is TX pause
- 3 enables full
dev.ixl.#.advertise_speed
Set the speeds that the interface will advertise on the link. dev.ixl.#.supported_speeds contains the speeds that are allowed to be set.
dev.ixl.#.current_speed
Displays the current speed.
dev.ixl.#.fw_version
Displays the current firmware and NVM versions of the adapter.
dev.ixl.#.debug.switch_vlans
Set the Ethertype used by the hardware itself to handle internal services.
Frames with this Ethertype will be dropped without notice. Defaults to 0x88a8, which is a well known number for IEEE 802.1ad VLAN stacking. If you need 802.1ad support, set this number to any another Ethertype i.e. 0xffff.
INTERRUPT STORMS
It is important to note that 40G operation can generate high numbers of
interrupts, often incorrectly being interpreted as a storm condition in
the kernel. It is suggested that this be resolved by setting:
hw.intr_storm_threshold: 0
IOVCTL OPTIONS
The driver supports additional optional parameters for created VFs
(Virtual Functions) when using iovctl(8):
mac-addr (unicast-mac)
Set the Ethernet MAC address that the VF will use. If
unspecified, the VF will use a randomly generated MAC address.
mac-anti-spoof (bool)
Prevent the VF from sending Ethernet frames with a source address
that does not match its own.
allow-set-mac (bool)
Allow the VF to set its own Ethernet MAC address
allow-promisc (bool)
Allow the VF to inspect all of the traffic sent to the port.
num-queues (uint16_t)
Specify the number of queues the VF will have. By default, this
is set to the number of MSI-X vectors supported by the VF minus
one.
An up to date list of parameters and their defaults can be found by using
iovctl(8) with the -S option.
Result 1
Interface lagg0_vlan2
Start Time 2022-07-04 20:12:10 -0500
Port 27449
General
Time Tue, 05 Jul 2022 01:14:37 UTC
Duration 30
Block Size 131072
Connection
Local Host 192.0.2.1
Local Port 27449
Remote Host 192.0.2.23
Remote Port 42218
CPU Usage
Host Total 78.54
Host User 14.81
Host System 63.75
Remote Total 0.00
Remote User 0.00
Remote System 0.00
Performance Data
Start 0 0
End 30.000273 30.000273
Seconds 30.000273 30.000273
Bytes 0 36067520404
Bits Per Second 0 9617917918.01361
