from the well,-that-was-fun dept
From there, the hacker got into part of the blog admin (don't want to get into too many details of how the blog backend works, but it actually involves two separate admins -- which are separate from other stuff we do). Then, he basically had pretty good access to doing some stuff (though not everything) on the blog. He poked around a bit, deleted a bunch of comments, deleted a whole ton of old story submissions (most of which were junk anyway -- so thanks!) and then replaced a few stories on the front page with his fancy "hacked!" claims.
After that, the story is pretty straightforward. Once we realized what happened, we put the old stories back in place and made sure to quickly toss up some more secure walls to keep him out of the admin. We also shut down comments and submissions for a while, even though we were pretty damn sure the vulnerability wasn't there (it wasn't), but we wanted to make sure. Then a few of us spent some time digging around to understand just what the guy did so we could retrace his steps and make sure we killed off the basic vulnerabilities. Considering that he tried to hit us from a bunch of different angles, this took a bit longer than expected. But, once we figured out the basics, it was just a matter of tracking down the actual holes in the code. It was a little frustrating, since we really thought we'd blocked out SQL injections -- but in the end, it turns out we didn't do it absolutely everywhere. Anyway, there's a fair amount of code to go through, so we've been going over it with a fine-tooth comb, and checking it twice, then locking it down again.
Finally, we've been restoring the lost comments (we're doing that right now, so they might not all be back yet), of which we believe we didn't lose any (there's a small chance that a very very small number of comments were lost). Restoring the lost submissions is a bit much at this point (as I said, most were junk anyway), so if you submitted stories late Friday or Saturday, and really think we should see them, perhaps submit them again.
On the whole, there's not that much to say, other than check your code carefully, folks. If there's a hole somewhere, eventually someone's gonna find it. Luckily, this guy didn't do much damage -- just a bit of vandalism -- and he kept a few of us from enjoying what had otherwise been quite nice weekends with our friends and families. But he got us to go over our code pretty carefully (and mentally kick ourselves a few times), and get in touch with our inner CSI detectives to track down exactly what happened.
Update: Well, that was just great. Less than half an hour after posting this, our network provider went down for nearly two hours, despite supposedly having all sorts of redundancies. It had nothing whatsoever to do with the hack, but was a bigger issue for the provider. However, it did slow down us restoring the comments, meaning that comments need to remain off for probably another few hours. This has really been a fun weekend.
Update 2: Comments are back. We did end up losing a few comments, mostly those right before the hack. Really sorry about that. If you said something really important and it's missing... say it again, please.