finished rewriting a lot of the BoredOS netstack
Hey reddit!
I've spent the last few weeks rewriting the network stack for my hobby OS, BoredOS.
Before i start yapping about what i changed, i'll quickly explain the netstack. BoredOS uses the lwIP stack to handle IPv4, Ipv6 (experimental), TCP, UDP and ICMP. The nice thing about LwIP is that i only have to supply hardware driver ring buffers, VFS fd's, some splinlock synchro, socket wait queues and a few syscalls.
Previously i was using a custom sysfs file interface to handle network traffic.. Which was quite shit imo. It didn't fit standard C lib calls or allow multiplexing network I/O alongside regular files. I replaced this monstrosity with a BSD socket layer and standard POSIX system calls like: socket, bind, connect, listen, accept, send, recv, select, poll, sendmsg, and recvmsg. Wiring sockets directly into the VFS fd table allows read/write and close to work on net sockets the same way they do on regular files and pipes.
BoredOS previously also had DHCP and DNS baked into the kernel with some very messy way of interacting with them, these have now all been moved into userspace with my own versions of standard fbsd netutils.
For packet handling, i moved processing out of the hardware interrupt handlers into a dedicated kernel worker thread. Processing packets directly in interrupt context was increasing interrupt latency and causing frame drops in heavy bursts, therefore hardware ISRs now just dump incoming frames into ring buffers and wake up a dedicated kernel worker thread via a wait queue. The worker thread sleeps when idle, wakes up on IRQs to drain the buffers into lwIP, and manages background timers for TCP retransmissions and ARP timeouts. For socket storage, i added kernel ring buffers backed by spinlocks and wait queues to handle blocking and non-blocking backpressure properly.
To allow my netutils and custom protocols to run without needing to ensloppify the kernel, i wrote a virtual TUN and TAP device driver. It supports layer 3 IP tunneling feeding into lwIP and Layer 2 Eth frame mode through TUNSETIFF ioctls. I also added raw socket support with AF_PACKET so userland packet sniffers can capture and inject raw ethernet frames directly. Of course meant for the skids to larp that they're hacking something xD.
For device management i added BSD style interface naming like em0 for intel gigabit cards, re0 and re1 for RealTek cards, vtnet0 for VirtIO, and tun0 for virtual devices.
With all these changes i also managed to speed up the stack by a LOT. Previously i was limited to a few Mbit/s and i am now able to push 800 Mbit/s in QEMU emulating an E1000 nic (maybe VirtIO is even faster? too lazy to test though) which basically maxes out the host machine connection.
EDIT:
Forgot to attach the gh:
https://github.com/BoredOS
