Monitoring with collectd

A while back I started graphing a few things to keep an eye on them, using rrdtool because change is hard. I then started switching to Grafana by way of ElasticSearch, until I realized there really isn’t a safe way to expose some Grafana graphs to the public, and on the off chance I want to look at something while away from home I like having the graphs exposed to the internet.

So I reactivated a couple of my graphs, but left a couple of others broken when I deactivated the ElasticSearch poller. Sometime many months ago I found collectd, which seems to do exactly what I want - instead of a bunch of PHP, Python, and BASH scripts stuck together with hot glue and chewing gum to feed my data into rrdtool, this program purports to do it all for me. Neat!

There it stayed, in a tab on one of my machines, for months, until yesterday when I didn’t really have much to do (I had physical things I wanted to do around the house, but I’m feeling run-down lately and decided to procrastinate on those).

Setting it up proved to be a bit of a pain in the arse - none of the docker containers appeared to do what I wanted (they all seemed related to something else, ie they had requirements for mysql or other applications I did not want), and the apt version in Ubuntu wants to install a pile of X11-related libs (there doesn’t appear to be a collectd-server package that doesn’t do this).

I finally found a fairly outdated, but still working container: fr3nd/collectd which mostly did the job. I don’t use privileged mode because I do not care about having /proc available, but everything else seems to work how I want it and I can’t spot anything obviously malicious in the dockerfiles.

Well, almost everything - it lacks the lm-sensors binary so I can’t grab temperatures yet. I’m also still monitoring the SMART temperatures with a bash script from the host, but I’ll try sort that out too.

This one’s fairly outdated, so I think I’ll try roll my own container at some point and include the sensors binary as well, and I may run it from Alpine instead of Debian. It looks like an interesting project, but I ran out of weekend (note the rather late timestamp of this entry).

For monitoring the NTP server (which I was surprised to find works with the antenna in the garage under a colorbond roof!) I am just using the built-in NTP plugin. I had to re-enable mode7 on my NTP server (not really a problem because it’s not public) and it does not monitor jitter, only the offset and “dispersion” which I don’t understand well enough to make use of, so am not graphing it yet. It’s on my todo list to put collectd on the timeserver as well, so I can graph other things like temperature and so on, but it’s a PCEngines APU running NanoBSD and I appear to have blown away the VM I used to generate it, and naturally did not document it at all, so I shelved that idea pretty quick.

Still though, the plan is to write a collectd plugin for the inverter (assuming that happens before I replace it with a bigger one) so I’ll have to cross that bridge eventually, but once again, I’ve run out of weekend so it’s time to stop.

But at least now all four graphs on my status page are populated with data again, even if I still have a lot more work to do to make it useful.

Update 2021-11-08: I created my own docker container, which I’ll publish later. It includes the sensors package, is built off Alpine, and has collectd 5.5 on it, which means I can use other stuff like the SMART module and so on.

I then got stuck into trying to programaticcaly generate the arguments for rrdtool graph with awk and got stuck with my bash script eating the spaces and quotes inappropriately, so I think it’s probably time I stopped fucking about and used the python module for rrdtool instead, and figured out a nice way to express how to generate the graphs. I can see this project spiralling out of control, but I’m actually enjoying it so far.

Horsham, VIC, Australia fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry

Happy Halloween! (modulo pandemic)

It’s Halloween, arguably my favourite holiday and with two familial exceptions and possibly the exception of the Chalupa just about the best thing to come out of North America. Unfortunately, we’re mid-pandemic again, which means this poor child is screwed again. We were discussing it today, and I don’t think he’s ever had a good one - his very first halloween, as an infant, was freakishly cold and we headed home early. His second, we were all teed up and then his pediatrician decided that because he was underweight, he must go to Riley Hospital in Indiana, immediately, no it could not wait until tomorrow, head there now or we call CPS (and they actually did, in what may have been the world’s most idiotic CPS case where even the caseworkers thought it was stupid).

Not long after that we left to come to Australia, where they don’t really do Halloween at the best of times but doubly-so in rural areas where my folks lived, and we were touring around for a lot of it. We moved to Horsham and for a time they were trying to make a go of it, with folks getting together and organizing a map, but the houses being so sparse that driving around was a must. Then this bloody pandemic started!

So we decided that regardless of what happened, trick-or-treating was off the table this year due to the pandemic (just as well, as it was a bust), so we just got a small bucket and some candy to put in it and planned to just make Duncan trick-or-treat his own house. Then he had a friend over, so we grabbed another bucket so they could both do it.

It ended up being a pretty good weekend, Sabriena wanted to bake a cake and the kids ended up helping, and then by the end of the night Sabriena had basically started digging out the boxes of Christmas decorations (her favourite holiday by a wide margin).

We’re still in negotiations for the degree to which we’ll skip Thanksgiving this year - I want my goddamn yams and pumpkin pie!

Horsham, VIC, Australia fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry Newer Entry

TV: Mounted

Ages ago I built a DIY-built entertainment centre, wrapped in two K-mart shelves, suitable for housing “all” my game consoles and the associated paraphernalia. It was built specifically to fit a wall in a rental house we lived in at the time (Shirley St), and when we moved out of that house into Pioneer Ct it would no longer fit.

So we modified it slightly, but it became quite the issue. You see, it no longer fit all my consoles (we doubled up some of them, but even then I was out of space), and the original plan was to get an RGB SCART switch so that all the consoles could be connected at all times, just ready to flip the switch and play. The problem with this is that a good quality SCART switch with sufficient inputs is not cheap. I could convert everything to BNC and buy something like an Extron Crosspoint, but even those are expensive in Australia, so in the end I just left the OSSC connected to the TV, and then when I wanted to play something I embraced the ritual of connecting it up.

I actually like this approach, because frankly it’s how a lot of my childhood went. We shifted the NES and the SNES around the house a lot, and some of my best memories are taking it to my cousin’s house or my great aunt’s house, and so a lot of the time connecting it up and flipping the TV to the right channel was just part of the process.

When we moved into this house, this big gloss-black beast didn’t fit the room at all and I was deeply unhappy with it the entire time. Sabriena had grown to dislike it too - I painted it black to make up for the fact I wasn’t confident in my ability to create a nice finish on it, and the black surfaces ended up collecting dust like no one’s business, with the pockets for all the consoles making dusting them difficult too.

So our plan from the get-go was to wall-mount the TV, put some light shelves around it with plants and knick-knacks, put the consoles elsewhere for storage and then I would connect them up when required. This weekend we finally got around to putting part one of that plan in place, using floating shelves (in “Ash Oak”) from Hammer Barn. We put four 900mm shelves around the TV and one 1200mm shelf under it. Everything’s anchored into the brick solidly and I’m reasonably confident it’s not going anywhere, but it remains to be seen whether the 10KG load limit for each of the shelves is going to prove enough, we may rethink this and go with something with an exposed support instead.

I’m absolutely rat-shit after doing this, so it’ll be some time before we move on to phase two. But at this stage, the plan is to remove one brick behind the TV, and knock the mortar off the back of the bricks to make a channel in the roughly-half-inch void between the two layers of brick. We’ll then have Baz the electrician come out and put a power socket back there, along with a brush dustcover so I can send a few long HDMI cables across the ceiling.

We’re going to put the consoles on the shelves on either side of the bar area, because frankly we’re never going to have sufficient people over where a bar becomes necessary and we’re just not sure what else to do with that space. So I’m going to put some HDMI sockets there, and some power (there’s already power on both sides, but not in a convenient place), and then tuck all the consoles away in there. I’ll put the scaler over there, and then I should be able to put things like the PC Engine (which in Japanese fashion has a really short controller cord) on the bar behind the couch and sit comfortably to play, instead of plopped on the floor.

But yes, for now it’s recovery time… drilling 34 holes into brick with a hammer drill took it out of me!

Horsham, VIC, Australia fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry Newer Entry

Monitoring the solar inverter

When we bought this house, it came with what we’re pretty sure is a 2kW PV system on the roof. I’m quite sure it’s old, and not optimized for our use: the instruction manual with it references software running on Windows XP, and it’s pointed westward (ie, optimized for afternoon use, when someone gets home, rather than all-day if it were oriented north). I want to replace it with the absolute biggest system we can reasonably fit on our roof (I’m hoping for something around 15kW), to maximize the potential for batteries some day and to hopefully eliminate our daytime energy use on all but the worst days.

But for now, it’d be nice to keep track of generation better than the utility company telling me what we sold them. Some time ago I worked out that there’s a DB-9 serial port on the bottom of the machine, and lots of folks have spent some time looking at the dodgy proprietary protocol the unit speaks. It’s documented in a PDF here (I’ve mirrored the PDF in case this goes down): https://www.snafu.priv.at/mystuff/growatt.html

A while back Greg gave me some serial cables that’d do the job for connecting to it, but I haven’t done anything with it until today, when I decided to take a look. I didn’t have anything with a serial port close enough for the cables to reach, so I dug out an old APU1c and connected it up.

This APU unfortunately only has FreeBSD 12 on it though, in a NanoBSD configuration (making getting additional software on it problematic). No Python, no Perl, no compilers of any sort. Only the base tools, stty, cat, some shell builtins, and cu which provded to not be much use. Worse still, I didn’t actually know how to drive tcsh or sh, I only really speak bash (not installed on this machine) these days after working at a Linux shop for way too long, so Peter on IRC had to come to my rescue with how to actually echo hex codes as the raw bytes.

After some messing around (getty was hogging my serial port), I finally got it to work with the most rudimentary tools:

stty -f /dev/ttyu0 speed 9600
echo -n $'\x3F\x23\x7E\x34\x41\x7E\x32\x59\x31\x30\x30\x30\x23\x3F' > /dev/ttyu0
echo -n $'\x3F\x23\x7E\x34\x42\x7E\x23\x3F' > /dev/ttyu0

and then watching the output:

cat /dev/ttyu0 | hd
00000000  02 0c 89 08 dc c3 f7 57  09 ec 0e c1 00 00 09 6b  |.......W.......k|
00000010  13 87 4e 22 01 7d 01 00  00 00 00 00 00 3c 00 02  |..N".}.......<..|
00000020  0c 89 08 dc c3 f7 57 09  f2 0e c1 00 00 09 62 13  |......W.......b.|
00000030  87 4e 22 01 7d 01 00 00  00 00 00 00 3c 00 02 0c  |.N".}.......<...|
00000040  89 08 dc c3 f7 57 09 f8  0e c6 00 00 09 6b 13 87  |.....W.......k..|
00000050  4e 22 01 7d 01 00 00 00  00 00 00 3c 00 02 0c 89  |N".}.......<....|

The output came much faster than expected - I requested a 1000ms (ie 1-second) update rate and it was spewing data constantly, not sure why that was. I’m also not sure how to stop it, so I rebooted the inverter by throwing both isolator switches to shut it up.

But picking out one of the records (starting at byte 7), it looks correct to me:

57          - start of record
09 ec       - PV1 voltage x10: 0x09ec is 2540, or 254V.
0e c1       - junk, ignored
00 00       - PV2 voltage x10: I do not have a second string.
09 6b       - grid voltage x10: 2411, or 241.1V.
13 87       - grid frequency x 100: 4999 or 49.99Hz
4e 22       - output power, watts x10: 20002 or ~2kW
01 7d       - temperature x10: 381, or 38.1C
01          - status, guessing 1 is normal?
00          - fault code, 00 == normal?
00 00 00 00 - four bytes of junk
00 3c       - EToday, x10: 60 is 6kWh?
00 02 0c 89 - EAll, x10: 134281, or 13428.1kWh, matches what's on the display.
08 dc c3 f7 - Seconds operated, 148685815, or 4.71 years

If the seconds operated only ticks up during daylight hours, as I think it does, that makes this quite an old system indeed… we may be eligible for some sort of grants to upgrade it. I am not banking on this though, and expecting to pay full price for it, which is why it has not happened yet.

So this will definitely work, it should be fairly easy to write a Python program which will read the serial port, blocking if necessary with a timeout, and every say, 5 minutes, sending the re-init command if the port isn’t communicating (because each time the unit goes to sleep it will need re-initializing). If it’s communicating, read these records, parse them and grab EAll, and feed it into RRDtool. That should be job done.

The person who published the datasheet has written a program in perl to do it: http://www.snafu.priv.at/mystuff/read-growatt

I’m not averse to perl (a long time ago I wrote quite a lot in it) but it doesn’t quite do what I want and I think I’ll have an easier time in Python. We’ll see.

I did find this software, which I don’t think does what I want either: https://github.com/lemval/growatt-rs232-reader - it exposes an HTTP server that I would then have to poll to get the data into RRDtool anyway.

Horsham, VIC, Australia fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry Newer Entry

Beanie: Walking Success

One of the rough parts about getting Beanie when we did is that in the middle of a pandemic lockdown, the obedience club aren’t able to operate (with outdoor participant limits being well below what they need to break even). We paid $120AUD for a “puppy preschool” at the local vet hospital, which basically only taught us a small handful of things we didn’t already know, but I was happy to spend the money just to have the ability to socialize the dog a bit, which went rather well, for the most part… but only lasted four sessions, and was itself interrupted so there was some delay between the first and last sessions.

Possibly as a result, and possibly just due to her personality, she’s very reactive to other dogs. Not aggressive, just misbehaves, barks, and jumps around a lot when there’s another dog within a very long range (sometimes as far as a hundred feet or more). Consequently, taking her on walks is quite problematic, we never get very far from the house before her behaviour turns to shit and we must head home.

Sabriena’s been working with her a lot to try and curb it, particularly with her barking at the school kids going past on their bikes, but it still didn’t really feel like we were getting anywhere.

We saw a recommendation from someone to drive our car out to the dog park and sit with her in the car, rewarding her for behaving herself, and then over the course of a few sessions moving the car steadily closer, then getting out of the car, and so on. This sounded pretty good so we had a go at it - naturally, we couldn’t find any damn dogs at any of the local parks on the first day!

Then today, we decided to try a different tack - driving out to the other side of the river, and going walking along the “rather bush” Langlands Track, which not that many folks use. We saw maybe four people the entire time, and each time we were able to get her well off the track and work on her behaviour. But you can see the results below:

Beanie's first real walk

Forgive the freedom-units because Sabriena’s a recovering American - this is a pretty fair effort, and at the end of it she was ratshit. We went home, she napped pretty well, hung out for a bit to watch TV with Sabriena and Duncan while I worked, then slept all night.

Horsham, VIC, Australia fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry Newer Entry