fwaggle.org

As I wrote about earlier, I spent the better part of the last couple of days trying out various tricks to soup-up WordPress websites. One of the things I managed to do was re-imagine the thumbnailer common to a good portion of those “premium themes”, to allow it to serve mostly-static content instead of relying on PHP all the time.

The results are pretty freakin’ awesome – based on some rough benchmarks I’m expecting somewhere between ten to fifty times as much users during load when it’s teamed up with nginx.

Of course it has some serious caveats, not entirely unlike the side effects of diet pills (pooing out fatty jelly isn’t entirely out of the question) and it’s rather annoying to have to edit your theme every time you want to change your look… but I think the results are worth it.

I’ve released the instructions for Super-caching TimThumb on Hungry Hacker, it’d be awesome if it were somehow rolled into the main script.

Well, it’s no secret to anyone reading my blog that we’ve been experimenting with building dig-proof sites. Today, those experiments took a huge leap forward. It’s probably also no secret that I’m a bit of an Apache fanboy – I’ve been using it for well over a decade now, and I’m a little reluctant to let go of it.

However, Apache has to take a back seat for some of our sites now (at the time of writing, Hungry Hacker is the only one) as we’re giving nginx a test drive. The preliminary results are stunning – absolutely stunning.

At first I wasn’t convinced, because on my little home test server (a celeron with measly amounts of RAM) it appeared like both Apache (event MPM, tons of modules removed, all kinds of speed hacks) and nginx (out of the box from ports) both appeared to run out of steam around the same time.

A short time later I realized that while the benchmarks on both machines appeared to choke up around the same place, the box was a whole lot more responsive with nginx under siege. While Apache was able to keep up with nginx (something to the tune of 2-4% behind it), the shell on the machine was pretty unstable during it.

So I decided to give it a shot on a live server, with a live site (hungry hacker), considering there’s very little involved in building nginx at all. With a bit of config-fudging, I managed to build a reverse proxy out of nginx for dynamic pages, with super-cached pages being served directly via nginx. The results are like night and day (it should be stressed these are non-keepalive requests):

Apache: 684.73 [#/sec] (mean)
nginx+Apache: 10027.18 [#/sec] (mean)

Without gzip enabled (I’m working on some nginx-config-fu to selectively serve pre-compressed super-cache files to browsers that’ll accept it, so far without much success), the ~10,027 requests/second resulted an almost-saturated gig-e port.

If we can pre-cache the text-based data (average savings of ~65% or so on Hungry Hacker), I’m pretty optimistic we can host absolutely surge-proof WordPress sites.

Akismet’s still proving itself useful, blocking a ton of shit like lipofuze reviews and other diet pill crap along with the usual suspects of “I really enjoyed your blog! I will check back often” on what’s usually the most boring page on the site. The real irony was that a good proportion of the spam is aimed at Strykar’s article on screwing over spammers!

We’ve been hacking a lot on WordPress lately, spread out over various sites… and I really have to wonder who made some of the design decisions or if, like most of the PHP I (and a good portion of the world) write, some parts of it were never designed and were instead grown. The result is some really idiotic default behavior for which there’s no easy workaround when you want to do something remotely out of the norm.

Take, for example, using category stubs as part of your perma-link URLs. A good many advanced themes have “meta categories” – Arthemia, for example, which we’ve hacked up for Hungry Hackers, has two: “headline” of which the latest post is shown up the top left, and “featured” of which several are shown on the top right.

It stands to reason you’re going to add these categories first, and then as your site develops you’ll be adding more later on. Well, when constructing a perma-link URL for a post, if you’re using /%category%/%postname%/, the lowest ID category is chosen for the “main” category, whose stub will be used to craft the perma-link – and there doesn’t appear to be any way to change that.

Sure you could make all your categories first, and then the “meta” ones later – but what happens later on when you want to add another category to an established site? Nightmare category juggling, or having featured articles having the URL /featured/some-great-article/. The worst part is, if an article is in two categories and you try to access it via the stub of a different one, it 404s. So at some point you have a really popular featured article, it gets a shitload of back-links, then you move it out of the “featured” group for some reason – all those back-links are useless. They 404.

Even if you’re not using any “meta” categories, in my opinion this is still silly behavior. It stands to reason that in most sites that grow in a sane way, the highest IDed category is going to be the most significantly relevant category. Think about it, your site is small and you start out writing about Flowers, putting all your articles in “/flowers/”. Later on, you add two new categories for “indoor” and “outdoor” – it’d make more sense for the URL to be based off the more specific category… except of course you have that whole 404 thing going on if you so much as move a post to another category.

I started out hacking on the WP-core, it’s simply a matter of swapping two comparison operators in _usort_terms_by_ID() and the highest ID category’s selected. Of course this might break shit in future, and would cause a ton of URLs to 404 if I update the WordPress core at some point and neglect to re-hack the change. But it’s really quite an easy hack in wp-includes/category-template.php (swap the underlined operators):

function _usort_terms_by_ID( $a, $b ) {
if ( $a->term_id > $b->term_id )
return 1;
elseif ( $a->term_id < $b->term_id )

I reverted that though, because I didn’t want to have to remember to do it with every WP patch. The very first “click here to update” notice could spell a whole lot of un-indexed content in search engines…

In the end, I ended up just re-numbering my meta categories to 999 and 1000, and hoping I don’t ever create 999 categories to cause a conflict. It was simply a matter of removing all posts from those categories, then editing the SQL database tables.

Update the term_ids in wp_terms and make them match in wp_term_taxonomy, and you’re good to go. That is, until such a time as you end up making your 999th category, so perhaps I should have selected a number just a little higher? ;D

Since we’re hopefully going to be playing more than one or two games this summer, I started taking inventory of what’s required to get my paintball gun in tip-top working condition.

Since I was doing that, I decided to take the various parts of my old paintball pages from Hungry Hacker and update them to put on the new site. We decided to make a whole knew section for Paintball, and since OtterSC Customs’ domain has expired, I’m also going to be putting together somewhat of a “Spyder Bible” since I can’t really see too much in the way of sites like that at the moment.

It’s funny to see the neighbours’ reactions when I go out and test-fire our paintguns in the back yard. Especially the welfare trash across the street (though to be fair, some of them are stuck on social security disability, hopefully for good reason) – those jerks always glare at us for no reason.

So because our California DC is still full (desperately waiting on more racks to come online), RetroHive’s been sitting semi-idle. That is, Ryan hasn’t been peddling any more Digg submissions to try and keep arigeitsu from having to work too hard.

Thanks to copious amounts of SEO, back-links and directory submissions, it’s getting a fair few amount of organic hits… but most organic hits are pretty easy for a webserver to handle. With the exception of very specific circumstances (say, a football fan-site around the time of a big game) organic hits usually come in in the form of a constant trickle, compared to the tidal wave of an article reaching the front page of Digg.

We did as much in the way of optimization as is about possible, with arigeitsu being good for about 2,000 concurrent hits at a time. I could probably unleash Apache further, because the load is still quite low serving up all this content (particularly with aggressive caching from WP-SuperCache et al) but because arigeitsu serves dual-purpose of a part-time web server and also hosts Mumble clients, I’m more worried about port saturation.

If we fill up our port for even brief periods, during those brief periods Mumble will be pretty well unusable. That means, per Sabrienix/MumbleDog’s SLA, I hand out a ton of credit. We just don’t want that to happen.

When the new machine dedicated for RetroHive comes online, I suspect we’ll be looking at around 6-7,000 concurrent hits without issue… but that’s really just a guess because I’ve never had to optimize a server for such traffic – the only high-traffic optimization I had to do was when I helped admin a download mirror for a PlayStation emulator and all we had to do was push out raw data.

So I’m excited to see how well we can tune this new box to take multiple Digg/Reddit/Slashdot attacks and maintain 100% uptime!

So I’m playing around with the 3.0 dailies of WordPress and… well.. it’s a little underwhelming. I’m not sure what entirely is going on under the hood that might be spectacular, but the only ground breaking stuff I’ve seen so far is the new default theme.

Sure, “Twenty Ten” is a pretty nice improvement. Kubrick is like a squirrelly little emo kid, albeit a little pretty but overall skinny and poorly justified. “Twenty Ten” is more like a meat head pumping human growth hormones and then hitting the gym – a glistening specimen of themely manliness if ever there was one.

Alright, I’m getting just a little carried away… but the new default theme is a pretty good improvement and will give beginning bloggers a great launchpad to get them started – simply throw away the header image and replace it (easy enough, with the new header editing features) with something more to your style and your blog will reek of your own personality. Not so with the original Kubrick, I’m afraid… Even if you edit up the image to something custom, it’s still going to look like it’s Kubrick.

On the whole though, I really don’t see 3.0 so far being worth a “major version bump” – it looks more like 2.10, but then again if you’re used to versions being a decimal number, that’s probably vastly more confusing. Decimal version numbers mean there’s pretty well nowhere to go but 3.0.

I’ll write more on 3.0 later as I play with more stuff on it. Don’t forget to check out Hungry Hacker and let me know if there’s anything broken on it (besides lots of 404s).

Our Linksys WRT54g was on the way out, and I didn’t want to wait to go buy another crappy little device from Walmart that’d just die as soon as the warranty expires, so I decided to make one of my home FreeBSD machines a broadband router again.

It’s been a while, but I quickly fell back into the groove and got it up and routing.. then I decided to go for a little extra challenge. First of all, I implemented UPNP on the subnet our PS3s are on, which means that online gaming on the PS3s “just works” without mucking about with firewall rules and such. That’s the subject of a forthcoming Hungry Hacker article, because some people seem to think that getting miniupnpd and a Playstation 3 talking is some kind of Voodoo.

Today, I decided to mess with ALTQ – a tool from OpenBSD for rate-limiting and prioritization, I figured I could implement my own form of QoS. Our ADSL is capable of 6mbps down, 768k up – which doesn’t go very far with things like BitTorrent running.

After some dabbling, a lot of reading, and several hours waiting for this stone-age Celeron to build a kernel with ALTQ enabled… I think I’ve got it the way I want it. For the sake of making sure the DSL had something left, I set the queues to 5mb and 700k respectively.

The upstream is split into three queues – gaming data at 100kb/s, interactive stuff at 100kb/s, and everything else at 500kb/s. Each of these queues can “borrow” bandwidth from the others if they’re not full and the bandwidth is needed (that is, they’re not capped to 100kbps), but they’re prioritized in that order.

The downstream stuff was harder. First, this machine also acts as a Samba/UPNP media server – so I had to split the interface into two queues one of 5mbps for the DSL and the other at 95mbps for LAN stuff. Again, they can borrow from each other, so if nothing’s coming in over the DSL we can get the full 100mbps via Samba if the disks will do it.

Then the 5mbps is split up, 500kbps for gaming, 500kbps SSH/IMs (aka interactive stuff), 2mbps for Web, 1mbps for streaming, and 1mbps for everything else. As with all the others, they can borrow bandwidth from each other (so if there’s nothing else going on, BitTorrent, even though it’s lowest priority, can still saturate the downstream).

Streaming is hard, because stuff like NetFlix goes over HTTP to port 80 – so it’s hard to figure it out from web content. We’ve been going through and adding the netblocks used for the CDNs for NetFlix et al to a table called “cdn”, which is then throttled as part of the “streaming” queue. It’s not perfect, but it works most of the time.

The real test? I jumped on Modern Warfare 2 and played a few games while steam downloaded Left4Dead at about 650KB/s… and it all worked great!

Now I get to sit on my headset and listen to dumbasses in dire need of acne products talking shit about how hardcore they are, without worrying about lag because something’s downloading in the background. If only there was a way to throttle them using only the power of Open Source.

So I stole Windows 7 for a day…

Not even a day really, more like a few hours. I managed to catch a virus on my PC, a particularly nasty one from a momentary lapse in judgment loading a website notorious for having malware in it’s ad providers in MSIE. Avast was no help, because apparently someone has some brand new malware and paid to have it included in an ad network. Boo.

So I decided since I still have Sabriena’s DX9-capable graphics card, that I’d give Windows 7 another shot on my desktop – and I installed the RC on it, only to learn shortly thereafter that the RC is on it’s way to expiry and forces shutdown every so often.

I really dig Windows 7, and performance is about on-par with XP, so I started looking up where to buy it. OEM is $99 for “home premium”, and students can apparently buy it for $69. Still a bit more than I’d like to spend for a five-year-old PC, so I kept looking and came across $35 “key only, digital delivery” licenses that looked legit. More digging, and I found what appeared to be legit keys being sold by people for as low as five bucks.

Here’s the deal with those cheap keys. They’re not 100% legitimate – sure they’ll activate and you feel like you got your money’s worth, but what they actually are in most cases is what’s called “TechNet” keys. TechNet is a subscription service Microsoft offers, where for $250/year ($350 for the first year) you can download and generate keys for most any piece of Microsoft software you like… the idea being developers can test their software on the full gamut of Microsoft installations.

The catch is you’re supposed to be the only one using them – if your organization has multiple people that do that testing, you’re supposed to buy a TechNet subscription for each person. It goes without saying that you’re not supposed to be selling these keys, given that you can’t even strictly speaking hand them off to so much as another person in the same company as you.

What will probably happen is Microsoft’s TechNet moderators will eventually realize this person generated and used a bunch of keys not in accordance with the TechNet license, and terminate the person’s TechNet account… no doubt invalidating the keys along with it. Just because you paid money and ended up with a key that validated when you installed it doesn’t mean you have any legitimate claim to use the software.

So after nixing that idea (I’m not about to spend money to steal Windows 7 when I’m perfectly capable of doing that myself) I found a loader to activate W7 with a random key… which seemed like a great idea, but then I realized I’ve basically got untrusted software running on my desktop… not something I really wanted to be doing. In addition to that, and the general uneasiness that revolves around pirate software in general, I also didn’t want my machine to suddenly fail to start one morning because Microsoft patched the crappy loader I was using… so I went back to my original plan of only using Microsoft products I bought via the “Microsoft Tax” and stuck my legitimate XP back on my machine. I wasted an entire day to do a reinstall of XP that should have taken 2 hours tops.

Oh, I missed one really cool part – Windows 7 supports Dolby Digital and DTS output, which I was thrilled to find out. Once I installed the proper driver for my on board graphics card and enabled the digital output, I was able to test both codecs via the Win7 sound control panel – the test pattern is awesome… way better than the white noise my receiver does. It does a massive drum sound out your sub, then rings a bell (or maybe it’s three different notes on a xylophone? Not sure) out each speaker. Glorious.

So I installed Left4Dead and tried to get digital output on it – again, no such luck. It seems getting proper surround out of Source engine over digital audio is nothing but hacks, you’re stuck with the same old analog audio or 2 channel digital audio as on XP.

Dejected, I played for about ten minutes… got accosted by a large fellow in dire need of belly fat reduction, then began the tedious process of reinstalling XP. Thus ends my less-than-one-day foray into the realm of pirate operating systems.

I just thought I’d let everyone know about those TechNet keys though, from a cursory Google search it’s not immediately obvious what’s going on and they look like they’re on the up and up – they’re not. If you don’t get a certificate or a sticker – with a hologram on it – then it’s not a legitimate key.

If you noticed my blog (or more likely, the Moodoo website) was going a bit slow here and there yesterday.. well it’s because my good friend Ryan’s website was getting hammered. I don’t think any clients were affected for too long thanks to our Nagios monitoring and swift action (despite the fact I was knee deep in a remodel at the time).

In case you’re wondering who Ryan is, he’s basically Sabrienix’s web design guy – long ago I discovered that while I enjoy doing web design as a pass time, I don’t care for it enough to get paid doing it (ironic really, that Sabrienix started life as a design/media company, and now neither of us want to do design). He’s also an SEO/web marketing genius, which his site RetroHive is a testament to – zero to ~100k monthly visitors in the space of a couple months is nothing to smirk at… sales plug: if you want this, give us a call.

So anyway, despite the fact our Apache server is severely optimized, we were worried about port saturation first and foremost… until it became apparent that php/fastcgi and timthumb.php are a terrible mix, despite the “caching” that goes on. I’ll be rewriting timthumb to make use of WP-SuperCache style mod_rewrite caching – that is if a thumbnail’s already generated I don’t want the PHP interpreter being loaded at all.

Our next step will be configuring a dedicated server for this site alone – if Ryan’s marketing skills in the first few months are anything to go by we’re going to need some industrial equipment to keep this site online through it’s growth. It’ll be great practice for Sabrienix – we’re pretty good at optimizing Apache, but there’s always more to learn… and as it grows it might reach the point where we need a cluster, which will be a great experience.

In the mean time though, we’ve neutered his site to try and keep the bandwidth levels down on the server until we arrange for it’s own server.