Up late again last night, dicking around. One of the primary issues remaining on our home network was the fact that I hadn’t stood up any of the LXD services, chief amongst them is Samba, which also provides backups for all our machines to the ZFS pool (where we then snapshot things, which should protect us against ransomware unless the ZFS server is compromised as well, which really isn’t in my threat model).
So it’s pretty important, so I spent a while looking at it last night, fully intending to - finally - move it over to K8s. I can’t recall what problems I had with moving it to Docker which led me to leave it on LXD, and of course I didn’t write it down either.
But the reason these services are down is that formerly my VM host straddled the two VLANs (work + home), so that the services were available on both. The reason for this was the inter-VLAN routing performance of the USG wasn’t great, but it does mean that everything on that machine is available on both networks. I could firewall it at the machine, but if it’s not immediately obvious from the last few blog posts I don’t tend to do things the easy way for some reason.
Anyway, I’m waffling - it struck me yesterday that since I use BGP to connect MetalLB for K8s onto my LAN, maybe I can do the same thing with LXD? The first thing to remember is that I have exactly one LXD host, so I could have done it with a static route, but again… easy way, pah!
The first limitation: BIRD (the BGP service on the router, my kingdom for an OpenBGPd port!) will flatly refuse to initiate a connection to a neighbor with the same IP as one that’s already active. This is probably sensible, since this is generally a misconfiguration, but in my case there really were two BGP hosts on the same IP (MetalLB initiates connections and LXD waits for them)! No matter, just configure an IP alias in my
netplan config to get an extra IP, and boom, done.
Next up, I need to move the LXD bridge’s IP range to something that doesn’t conflict with anything else on my network, and while I’m at it let’s turn NAT off since the boundary router will handle that:
lxc network set lxdbr0 ipv4.nat=false
lxc network set lxdbr0 ipv4.address=10.1.0.1/16
LXD itself needs to be configured to use BGP:
lxc config set core.bgp_address=10.0.0.3:179
lxc config set core.bgp_asn=65536
This tells LXD to listen to BGP connections, but after I configured BIRD to connect to it, it was dropping the connection immediately after the first BGP packet - you need to tell LXD to expect a connection first, and you do that at the interface level, not globally:
lxc network set lxdbr0 bgp.peers.router.address=10.0.0.1
lxc network set lxdbr0 bgp.peers.router.asn=65535
You can specify a password here too, but I’ve ommitted that for blogging purposes.
Oh, and one other thing - I’m not sure what I’ve got misconfigured or what’s going on, but I had to add the statement
interpret communities off; to the
protocol bgp section for LXD. I’ll have to research why LXD is sending something that BIRD doesn’t want, but without this I get
rejected by protocol in the logs. I could also configure BIRD to not send routes to LXD, but it’ll drop them unceremoniously anyway and it’s not like I do BGP for a living.
Now you can restart the containers (mine were all stopped), and they should have access to the internet and the router should have a single route to the LXD bridge’s entire subnet via the LXD host’s IP. The documentation for LXD’s BGP support suggests that each instance will have its own route, but I’m suspecting that’s only the case when you’re using a cluster, if there’s only one host it doesn’t make sense to fill up the routing tables with a million routes that all say the same thing.
Next up, I don’t want LXD’s dnsmasq to shift the IP around on me if I say, recreate the container, so I need to set a static IP for my LXD container. This is unintuitive, but it does work: you remove the interface from the
default profile, then manually add it back, and then it lets you override the
ipv4.address value with whatever you like:
lxc network attach lxdbr0 samba eth0 eth0
lxc config device set samba eth0 ipv4.address 10.1.0.3
lxc start samba
Finally, something that was specific to my network, but I’ll be kicking myself if I don’t document it here: when I first started the containers, they didn’t have a default route. I thought this was a BGP thing at first, but it was a leftover from the “straddles two VLANs” days. I had DHCP on both addresses, and wanted the default route to go out over a specific network, so I had the
lxdbr0 interface configured to send an empty router list. I did this by adding
dnsmasq.raw on the interface, but I had to remove that and restart networking on the containers to get the default route back.
Once that’s all done, all that was left was to add a firewall rule to allow machines on the “Home” VLAN to access these services and I’m all set.
Long-term, the plan is still to move the last of these services over to K8s, but this will still be handy to have if I ever want to use LXD as a KVM manager (I tried unsuccessfully to run Windows NT 4 on it).
With this in place, that means that the only things left on my list of shit to fix are:
- SSL cert on the router.
- A VPN on the off chance I travel for work (will likely use wireguard for this).
Over the last couple of days, I spent more time figuring things out on my modified EdgeRouter Pro. Part way through, I got bored and decided to install OpenBSD, just to be sure. Installation was very easy, and it was starting to look like we might be on to a winner - I did’t have the read-only filesystem I wanted to protect the flash, but others have already solved that problem. With
pf(4) disabled, we were pushing packets around at pretty-much gigabit speeds (not quite, but close enough) between iperf3 on two different instances:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 981 MBytes 822 Mbits/sec receiver
Unfortunately, the story is not the same with
pf(4) enabled (with just the default ruleset, nothing crazy and no scrubbing):
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 502 MBytes 421 Mbits/sec receiver
That’s… suboptimal. I don’t have gigabit internet currently (maybe one day), but what I do have is a Samba server on one VLAN and clients that need to access it on another VLAN, and at this stage the routing between them needs to happen at the router.
So it was back to OpenWRT (which due to installing a modified bootloader to boot BSD meant flashing back the EdgeRouter firmware, but it didn’t take long to do via serial console), I configured everything and started looking at getting it set up. I swapped the wireless clients over to it on Thursday night, which honestly caused more grief than it solved, but on Friday night I swapped everything over to it and unplugged the USG-3.
I spent a while figuring out how to get BGP working, in fact that’s what I spent most of Friday night doing. I started out with BIRD2 (the combination ipv4 and ipv6 version), but it crashed immediately on the
octeon architecture… a better individual would have made a debug build and tried creating a bug report, but I simply moved on.
Next up, installed FRR, and configured it. I gotta say, if you rewrite something like Quagga, and don’t take the opportunity to re-do the configuration file format, I really have to question your motivations. It might be that they didn’t want to alienate Quagga nerds, but from what I can tell those folks are all still using Quagga anyway so I’m not sure what’s gained. Anyway, after quite some time I was pretty sure I had FRR configured correctly… except it wasn’t routing. It was accepting the routes (Showing as
*> (active route) and
*= (perfectly fine route but not the active path)), but simply not injecting them into the kernel routing table.
I spent hours on this, before finally giving up and installing
bird1-ipv4, since I don’t need BIRD to handle the IPv6 portions of my network. I started out with the “luci” module for it, but since I didn’t really know what I was doing, this caused more harm than good and I learned that if the configuration is invalid, OpenWRT doesn’t seem to have any back-off on the start-up of this daemon, resulting in me OOMing the device several times before I gave up, blue away all the UCI configuration, and configured it by hand with a file like Dog intended.
But once I got that working, it didn’t take very long at all to make it work and MetalLB was announcing routes correctly and I was able to get to bed.
I still have a few more things I’d like to do with it:
- IPv6 basically doesn’t work at all.
- I’ll probably, since I have 8 ethernet ports to work with, split the VLANs out onto their own physical interfaces, to increase bandwidth at the router.
- None of the LXD services are back up, due to network changes. I’ll probably just move those services to K8s, because the BGP stuff for LXD doesn’t look super convincing.
- HTTPS on the router, probably via LetsEncrypt.
I have wanted a soldering station for the longest time - I enjoy doing hardware mods for my video games stuff, among other things, and for the longest time I’ve made do with trash-tier equipment because I could never afford anything better. Heck, since we’ve been in Australia, I’ve been making do with a cheap 50W Jaycar special with a fat tip, and nearly screwed up the audio mod on my OSSC because of it (and my shaky hands).
A popular YouTuber recommended these cheap KSGER things from Asia, and it’d been on my todo list for some time. They warm up quick, survive heat-dumps on large ground planes reasonably well (at least as well as a low-end Hakko for about 3x the price), the circuit inside seems sensible (though there’s no telling if they’ve changed it since), and they take fairly standard tips.
But I like instant gratification, so it was a toss-up… spend significantly more on the Jaycar thing that I can have now, spend about the same on a low-end Hakko from an Aussie seller and wait about a week, or wait 4~6 weeks for a cheap one to come from Asia. I thought I’d found a good compromise: an eBay seller in QLD selling one for about $40 more than the cost of buying one on AliExpress. Purchased!
In a deviation from my normal (click Pay Now and then press my nose up against the window waiting), I didn’t look at it for about two weeks, when I got an email from the seller: “it’s shipped, here’s your tracking information”. Not off to a great start, but when I clicked on it, the fucking thing’s coming from Singapore anyway.
So at this point I’m pretty livid, I paid a premium because the item was listed as being in QLD, so I pushed the seller for a please-explain, and didn’t really get a satisfactory answer. If I’d wanted to wait a month for it to come from Asia I would have just bought a cheap one! Plus, to make matters worse nothing had updated on the tracking for about two weeks, and some stupid form-email about COVID delays isn’t going to cut it when I’m this angry. So I waited until the last possible moment, and opened a case with eBay.
Luckily, it arrived today - I ordered it on the 19th of April, no big deal - so all’s well that ends well I suppose, but I’ll refrain from leaving feedback for this sonofabitch too because I’ll have nothing nice to say and their retalitory feedback might ruin my 100% positive rating.
I’ve been toying with the idea of de-Unifiing our network for a while, but there’s really precious little that’s a drop-in replacement for it that doesn’t have some extreme drawbacks. But after much research, I came up with a handy stopgap solution: with the exception of switches (not sure why), I can run third-party firmware on my Unifi hardware to get off the ecosystem.
I started looking around at what my options were, and found a few articles of folks running BSD (Open and Free, though Open seems to work better) on the USG3 gateway. But doing this means taking everyone offline during the process, and if I brick it then we’re offline permanently until a replacement comes. I figured a quick check to see if I could find a bricked USG3 for cheap, and had no luck… most of them functioned, and thus seemed to go for about $150AUD. But there was a USG-Pro for about that price bidding, so if I was going to buy a backup then flog my old one later, why not upgrade?
Why not indeed, so I watched the auction to see where it went, and it quickly went well out of my price range. Indeed, it finished up at practically brand-new price and I have no idea why.
I did however manage to find something similar: the EdgeRouter Pro 8, which is effectively the same hardware, but without the fancy Unifi firmware… but I’ll be trashing that anyway. So on Thursday after much deliberation (I briefly considered buying a 1U amd64 machine instead but it too was quite expensive and probably heavier on power), and today it got here.
So after work, I set about building a serial cable for it (to avoid repeating the “wires poked into the pin sockets of a null modem cable” from getting the UPS going). I then downloaded OpenWRT (because the installation looked easier than the OpenBSD one, and performance is probably better), created a USB for it (which led to a whole other thing), booted it up and installed it. I literally spent more time soldering together the cable (a smarter individual would have simply ordered one earlier).
After some getting used to figuring out how to configure it, I have two different LAN interfaces, not on the same bridge, discrete interfaces on different subnets. I ran iperf3 on my desktop and laptop and got wire-speed routing between the two. That’ll do nicely!
I’m still lost with OpenWRT, so it’s tempting to try OpenBSD. I’m suspecting that with
pf enabled, it won’t perform quite as well as OpenWRT does with the hardware offloading support, but I might give it a go anyway. On the other hand, if I put OpenWRT on the WAPs, I’ll have to learn how to work it anyway, so I’m not sure which way I’ll go yet.
I’ve got a laundry list of things still to configure before I can put it into service:
- VLANs (I think they’re configured correctly)
- Subnets on the VLANs (I think I’ll take the opportunity to rearrange them, but it wouldn’t take long to configure the same ones I have now)
- Firewalls between the VLANs
- Local DNS
Plus some other “nice to haves” that I couldn’t be bothered making work on Unifi:
- Pihole, or some other sort of malicious domain blocker.
- Wireguard, instead of L2TP that I set up via Unifi.
Anyway, because it’s not really an EdgeRouter anymore, and I had the labeller out anyway, I’m calling it the FudgeRouter - Fraggle’s EdgeRouter, but with a kiwi accent.
Today, I needed to build a bootable USB for something, so I used my Linux-powered MacBook Pro to do so. In the process, I needed
wget, so I did
pacman -Syu wget and it took longer than I’d appreciate because I had about a week’s worth of package upgrades to get through.
After making the bootable USB though, I decided to reboot, because quite often X crashes after a lot of upgrades, so better to preemptively strike… and when I rebooted, it wouldn’t boot - init failed to launch!
I figured I’d accidentally misfired at some step and blown away my filesystem or something… it wouldn’t be the first time I’d done that, though normally it’s with
dd and not
newfs, but hey, shit happens.
After a while of fucking about, I managed to get Alpine linux loaded from a USB and take a look at the filesystem. All my stuff was intact! It looked like the initramfs image was corrupted somehow, most likely due to the package upgrades and not my ham-fisted filesystem twiddling (a cursory Google search suggests this isn’t unheard of).
But how to fix? I found suggestions to use
mkinitcpio, but doing it from a USB live-device wasn’t going to work, until I stumbled upon the missing ingredient:
arch-chroot. I’d already tried a regular
chroot to get into my main root filesystem, but it complained about
/proc (easily solved) and
/dev (not so easily solved) not being mounted.
arch-chroot takes care of all this for you.
So the solution was:
- Boot from arch installer USB.
mount /dev/sda3 /mnt
mkinitcpio -p linux
I thought I was going to have to
rsync everything off and reinstall!