Power outage!
I had just finished making a coffee, and Sabriena was gearing up to trim the dog’s nails when right as she went to turn on an audiobook on her headphones, one of the speakers in our house made a loud noise similar to the bluetooth pairing noise. It thus took us both some seconds of confusion before we realized the power had gone out.
This is not actually - touch wood - a terribly common occurrence out here. We’ve had one large-scale outage in the 12 years or so we’ve lived in Horsham, and that was when the local substation caught fire on a very hot afternoon. We’ve had a couple of smaller ones, 20 minutes or so, usually involving either a storm or a car accident with a power line, all of which the UPSes coped well with and we went right on with our business.
Today’s outage however appeared to be one of the larger ones - the outage map looked rather huge and listed the location of the outage as the substation (not sure if they all do?). I was in the process of shutting down machines (it shows how infrequent these issues are that I haven’t automated that) when the rack UPS suddenly died. It looks like it’s definitely time for more batteries (I have been saying that for a while now though):
It did illustrate that when they swapped out our NTD, I neglected to plug it back into the UPS… so I did that, and we had internet for about 20 minutes before that UPS died also. It’s a 1500VA Cyberpower model, so it’s not surprising it doesn’t have a huge amount of juice in it, but that did seem premature as well, and it’s a few years old so it may want a new battery too?
The power came back about 30 minutes later, the disk server gave me just enough grief starting up that I had to dig out a 4:3 monitor in anger, but it actually started up without help if I was just a little more patient.
Getting all the services to come back online required a bit of work - for some reason my Kubernetes cluster will start them without the requisite iSCSI or NFS mount working, they’ll crash repeatedly, and I’ll have to delete and recreate the pod and sometimes the entire deployment before it’ll quit doing it and act right.
Unless we get a few more of these, I don’t think it’s quite worth dumping money into a house battery yet, but I am going to have to grit my teeth and order replacement batteries for the UPSes.
fwagglePublished:
Modified:
Filed under:
Location:
Horsham, VIC, AustraliaNavigation: Older Entry Newer Entry