The problems started about a week or so ago when a nasty UTF8 exploit was released that let people crash the Murmur process – because we use real virtual hosting instead of that godawful TCAdmin script that was floating around, the unfortunate side effect would mean that if one Murmur blew up, they all would in the same location. Needless to say, we hop on security issues ASAP to make sure that doesn’t happen.

We got that fix all built out, tested it, and then placed the updated binary in place of the old one so if someone did blow up our Murmurs, they’d restart impervious to the attack. We figured that’d be better than just restarting at an arbitrary time and pissing someone off (because there’s never a good time to restart a bunch of voice servers), and it’d give us time to look at the second exploit.

The second one’s particularly nasty. It looks like it’s a bug in QT’s QSslSocket, and indeed that’s what most everyone’s billing it as – however it also looks like the bug is either fixed or mitigated by updating OpenSSL. I’m personally not clever enough to figure out where the bug is or how it’s fixed, all I know is that OpenSSL upgrades stopped the exploit that was in the wild from working (which anyone can download and point at any of the public servers and make them eat shit, so that’s not fun) so that was good enough for now until more information comes around.

After a lot of messing around with QT’s weirdness with regards SSL, we finally got it working so we restarted the servers with updated versions ready. The reason we restarted on purpose after this update is the second exploit doesn’t crash the server, it makes them go into an infinite loop, so they just sit there. Our monitoring systems would go crazy, but they wouldn’t actually restart the process because it’d still be running.

The downside to this was that the same upgraded OpenSSL that broke the exploit also broke public server registration. So our current servers that are up and running right now, none of them can be listed in the public server list. After a lot of messing around and hacking, we think we’ve gotten a completely bug free Murmur, but I’m running on very little sleep so we’re going to do a little more testing before we restart the servers again.

For what it’s worth, prior to these nasty bugs rearing their ugly heads, our servers have been up constantly for basically the entire year. We had a few spats of network issues here and there, but our datacenter staff have been taking the best vitamins and on strict health regimens to make sure they’re swift to react to them.

I might wait until monday or so, just to get past the “premium” gaming time, before we restart the servers. I mean yeah, it is summer, but those of us who work don’t particularly want to be in the middle of a raid or something and have our server reboot.

Say you write about getting shar pei dogs, there will be a post almost on topic and a link to say “wrinkle cream reviews” in the name/url section. Okay, maybe they’re not that badly off-topic, some of the comments and article subjects I’ve noted:

• VOIP Telephony links on posts about Mumble.
• iPhone on posts about Windows Mobile smartphones.

I’m sure there’s a few others but I don’t feel like going through the spam buckets on my other WordPress sites. The funny thing is they all manage to sneak past Akismet (probably because they’re all uniquely written), but they do all end in three random punctuation marks, like this:

Hi I liek ur phones I hope to one day own 1′-,

It’ll be interesting, given the wealth of keywords in this post, what they decide to try and link it to now.

For the longest time we did quite well with eBay’s affiliate marketing network via Commission Junction (it’s now called the eBay Partner Network and run in-house at eBay) – we realized we were never going to get rich but we made a few bucks a month we could spend on new projects here and there.

Then a while back, they switched from paying us on a per-action basis (so if someone bought a PS3 from one of our links we’d make a few bucks), to paying per-click on a quality basis. Most of the people who read our site are looking for the cheap way to do something, so very few of them actually bust out the ol’ Paypal account on eBay. In fact so few of them do, our quality of clicks has been stuck at zero for months, which is terrible because I know we’ve sent at least some quality traffic their way but we’re still stuck at zero.

So I’ve been thinking about adSense, because apparently it’s pretty fire-and-forget, assuming you don’t get the boot for fraud clicks. The only things holding me back is that I won’t have fine-grained control over what the links are, where they go, or how relevant they are (as I do with eBay) and it’d kind of be breaking our site’s creedo of not having “super-liminal” advertising.

So combine that moral dilemma with the fact I might either a) make no real money on it or b) get booted for click-fraud like so many other legitimate advertisers seem to be doing, and it’s pretty easy to see why I’m dragging my feet.

Can anyone forecast approximately what the income would be? We’re down to about 2k visitors and 3k page views per month right now, and we’re not tracking ebay click-through rates at the moment, but in the past it’s been between 1 and 5%. I’m assuming that as fancy as Google’s system is, it’s not as good at picking out relevant stuff as a real author doing so, but still.

Personally, I’m still not a huge fan of Open Source OSes for the desktop. Like I said earlier today, security notwithstanding most every other criticism of Windows just plain isn’t true any more… I haven’t had to do the daily-reboot thing with any NT-based OS starting from NT4.

For server OSes, FreeBSD’s my thing… a Daemon’s always going to be cooler than a tuxedo penguin, but really it’s all about the hier(7). For that reason, I really can’t conceive of why something like debian/kfreebsd exists – if anything I’d prefer the wide swath of hardware support covered by a nice clean BSD userland, not the other way around.

I seriously cringe every time someone goes “oh, you like FreeBSD? You should try Gentoo – it’s a lot like BSD”. Seriously, no… stop saying this.

To me it’s almost a case of “everything looks like a nail” with respect to jQuery – it’s one hell of a fat (cool, but fat) library for just ensuring a script is called without cache control. The same thing could be fixed with a transparent image, or a lightweight home-brew AJAX call-home without the entirety of the library included for every page view.

Disabling this plugin lopped off 25% of the average page’s uncompressed total file size – to be fair, with gzip compression on it was probably closer to about 5KB actually saved (though I have no idea how much jQuery weighs gzipped) but the fact is we saved another HTTP request, and a script request in the <head> of the page no less.

Median page-load times according to the PageSpeed plugin for Firebug look to be between 600ms and 1 second – I think that’s good enough, actually. I just need to think of something to put in the unoccupied territory of every page now, and then I should probably get back to work putting up content.

The results are pretty freakin’ awesome – based on some rough benchmarks I’m expecting somewhere between ten to fifty times as much users during load when it’s teamed up with nginx.

Of course it has some serious caveats, not entirely unlike the side effects of diet pills (pooing out fatty jelly isn’t entirely out of the question) and it’s rather annoying to have to edit your theme every time you want to change your look… but I think the results are worth it.

I’ve released the instructions for Super-caching TimThumb on Hungry Hacker, it’d be awesome if it were somehow rolled into the main script.

However, Apache has to take a back seat for some of our sites now (at the time of writing, Hungry Hacker is the only one) as we’re giving nginx a test drive. The preliminary results are stunning – absolutely stunning.

At first I wasn’t convinced, because on my little home test server (a celeron with measly amounts of RAM) it appeared like both Apache (event MPM, tons of modules removed, all kinds of speed hacks) and nginx (out of the box from ports) both appeared to run out of steam around the same time.

A short time later I realized that while the benchmarks on both machines appeared to choke up around the same place, the box was a whole lot more responsive with nginx under siege. While Apache was able to keep up with nginx (something to the tune of 2-4% behind it), the shell on the machine was pretty unstable during it.

So I decided to give it a shot on a live server, with a live site (hungry hacker), considering there’s very little involved in building nginx at all. With a bit of config-fudging, I managed to build a reverse proxy out of nginx for dynamic pages, with super-cached pages being served directly via nginx. The results are like night and day (it should be stressed these are non-keepalive requests):

Apache: 684.73 [#/sec] (mean)
nginx+Apache: 10027.18 [#/sec] (mean)

Without gzip enabled (I’m working on some nginx-config-fu to selectively serve pre-compressed super-cache files to browsers that’ll accept it, so far without much success), the ~10,027 requests/second resulted an almost-saturated gig-e port.

If we can pre-cache the text-based data (average savings of ~65% or so on Hungry Hacker), I’m pretty optimistic we can host absolutely surge-proof WordPress sites.

Akismet’s still proving itself useful, blocking a ton of shit like lipofuze reviews and other diet pill crap along with the usual suspects of “I really enjoyed your blog! I will check back often” on what’s usually the most boring page on the site. The real irony was that a good proportion of the spam is aimed at Strykar’s article on screwing over spammers!

Take, for example, using category stubs as part of your perma-link URLs. A good many advanced themes have “meta categories” – Arthemia, for example, which we’ve hacked up for Hungry Hackers, has two: “headline” of which the latest post is shown up the top left, and “featured” of which several are shown on the top right.

It stands to reason you’re going to add these categories first, and then as your site develops you’ll be adding more later on. Well, when constructing a perma-link URL for a post, if you’re using /%category%/%postname%/, the lowest ID category is chosen for the “main” category, whose stub will be used to craft the perma-link – and there doesn’t appear to be any way to change that.

Sure you could make all your categories first, and then the “meta” ones later – but what happens later on when you want to add another category to an established site? Nightmare category juggling, or having featured articles having the URL /featured/some-great-article/. The worst part is, if an article is in two categories and you try to access it via the stub of a different one, it 404s. So at some point you have a really popular featured article, it gets a shitload of back-links, then you move it out of the “featured” group for some reason – all those back-links are useless. They 404.

Even if you’re not using any “meta” categories, in my opinion this is still silly behavior. It stands to reason that in most sites that grow in a sane way, the highest IDed category is going to be the most significantly relevant category. Think about it, your site is small and you start out writing about Flowers, putting all your articles in “/flowers/”. Later on, you add two new categories for “indoor” and “outdoor” – it’d make more sense for the URL to be based off the more specific category… except of course you have that whole 404 thing going on if you so much as move a post to another category.

I started out hacking on the WP-core, it’s simply a matter of swapping two comparison operators in _usort_terms_by_ID() and the highest ID category’s selected. Of course this might break shit in future, and would cause a ton of URLs to 404 if I update the WordPress core at some point and neglect to re-hack the change. But it’s really quite an easy hack in wp-includes/category-template.php (swap the underlined operators):

function _usort_terms_by_ID( $a, $b ) {
if ( $a->term_id > $b->term_id )
return 1;
elseif ( $a->term_id < $b->term_id )

I reverted that though, because I didn’t want to have to remember to do it with every WP patch. The very first “click here to update” notice could spell a whole lot of un-indexed content in search engines…

In the end, I ended up just re-numbering my meta categories to 999 and 1000, and hoping I don’t ever create 999 categories to cause a conflict. It was simply a matter of removing all posts from those categories, then editing the SQL database tables.

Update the term_ids in wp_terms and make them match in wp_term_taxonomy, and you’re good to go. That is, until such a time as you end up making your 999th category, so perhaps I should have selected a number just a little higher? ;D

So I reverse the two tags, and change wp_title() to wp_title(‘-’, 1, ‘right’), which puts the separator on the right hand side instead of the left. Awesome!

Wait, something’s not quite right – the dash is a little longer than the one before “Mozilla Firefox”. I go to check another page – Google’s Webmaster Tools – and they’re using a plain ol’ hyphen and it looks the same. What the fuck is this? I check the source of my rendered WP blog, and it’s not a hyphen at all.

For some reason, if you set wp_title’s “sep” parameter to ‘-’, it arbitrarily changes it to an “en dash” (&ndash; or a dash the width of an “n”). Considering that the default is &raquo;, I can’t for the life of me figure out who the hell thought this would be a good idea.

Can someone explain it to me? If I meant to have an &ndash;, I’d have put one!

Since I was doing that, I decided to take the various parts of my old paintball pages from Hungry Hacker and update them to put on the new site. We decided to make a whole knew section for Paintball, and since OtterSC Customs’ domain has expired, I’m also going to be putting together somewhat of a “Spyder Bible” since I can’t really see too much in the way of sites like that at the moment.

It’s funny to see the neighbours’ reactions when I go out and test-fire our paintguns in the back yard. Especially the welfare trash across the street (though to be fair, some of them are stuck on social security disability, hopefully for good reason) – those jerks always glare at us for no reason.