Website outage!

I woke up this morning to an email from Keybase about my web-proof being deleted - it didn’t sound right, but I have moved some things around so I guess it could have happened some time ago and they’re just noticing it? Went to check the server out, and found a huge mess:

Everything 403ing for some reason
Unable to log in via SSH due to EPERM on authorized_keys.
Unable to log in via the console because I can’t paste root’s password and it has symbols I can’t manually type.

Fuck, no time to fix it, I have work to do!

Fortunately, I had a brief moment around lunch time to take a look at it, so I threw the machine into single user and set root’s pass to something I can type. Ran fsck on the filesystem, and it found some stuff, but nothing disastrous, and all the files are there, permissions look fine all the way up the tree, and yet I’m unable to su to my own user due to EPERM:

su -m fwaggle
/usr/local/bin/bash: Permission denied

At this point I’m pulling my hair out and went to IRC to ask for ideas, where pez came to my rescue with a forum thread - these things happen if the perms on /. and /.. are too strict, and sure enough, mine are set to 700 for some reason. A quick chmod and things are back to normal, but how’d this happen? I don’t recall chmoding anything yesterday, though I did use freebsd-update and upgraded all my packages also.

I’ll have to file that one away though.